Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unwrap is not stable when supplied with stable RNG #616

Open
RomanHargrave opened this issue Dec 28, 2024 · 9 comments · May be fixed by #617
Open

unwrap is not stable when supplied with stable RNG #616

RomanHargrave opened this issue Dec 28, 2024 · 9 comments · May be fixed by #617

Comments

@RomanHargrave
Copy link

I have been doing some signal analysis and happened to notice when applying unwrap in the below manner that it provides unstable results, even when applied with a stable RNG.

using STFT, LibSndFile, FileIO, Test, Plots
using DSP: unwrap

wk_data = Array{Float64}(load("anything.wav")[:, 1])

fft(sig) = 
	STFT.analysis(wk_data, DSP.hanning(64), 32, 192)

Test.@test sum(rand(StableRNG(10), 100) - rand(StableRNG(10), 100)) == 0
Test.@test sum(fft(wk_data) - fft(wk_data)) == 0
Test.@test sum(angle.(fft(wk_data)) - angle.(fft(wk_data))) == 0

ft = fft(wk_data)
ϕs = angle.(ft)
	
measure() = 
	sqrt(mean((DSP.unwrap(ϕs; dims=1:2, rng=StableRNG(10))
		     - DSP.unwrap(ϕs; dims=1:2, rng=StableRNG(10))) .^ 2))

es = [measure() for _ in 1:20]
plot(es, label="RMSE")

image

And so forth.

I am trying to measure phase information for comparison between samples, which requires the most (ideally completely) stable measurements as are possible. The error between applications is both significant and well-enough distributed as to propagate to other metrics,

Here is one measurement of a signal:
image

And another of the same signal - note the visually distinct changes to both the phase heatmap and the troughs at either end of the entropy measurement as well as the fact that the range of the unwrapped data is now (roughly) [-175, 75] as opposed to (again, roughly) [-25, 175] :
image

I am only noticing this fairly late, so have not scrutinized the implementation to see if this is to be expected, is caused by some issue with the wrapped phase information, or is an actual issue. I'll also accept the response that I am using DSP.unwrap in an unintended manner, or that I should not expect stability, should it include some better suggestion - I usually work much lower in the stack, where the most complex math tends to be memory alignment (humor attempt complete), so I am still becoming acquainted with some of the stuff here.

@RomanHargrave
Copy link
Author

Upon review of the unwrap code this afternoon, I determined that this is due to use of rng() inside of a parallel loop, which leads to race-like out-of-order reliability value generation.

@wheeheee
Copy link
Member

Possible, but I can't reproduce with

julia> measure(s) = ((Random.seed!(10); unwrap(s; dims=1:2)) - (Random.seed!(10); unwrap(s; dims=1:2))).^2 |> mean |> sqrt
measure (generic function with 1 method)

julia> ps = angle.(rand(ComplexF64, 1000, 1000));

julia> measure(ps)
0.0

@RomanHargrave
Copy link
Author

Be sure to start julia with additional threads - julia -t 8 or somesuch amount. When I start from the command line it usually only starts with one in the pool.

@wheeheee
Copy link
Member

Yes, that was run on a threaded session.

@RomanHargrave
Copy link
Author

RomanHargrave commented Dec 29, 2024

It looks like this is down to the semantics of seed! and the implementation of whatever the default RNG is - I am going to guess the RNG is cloned from its starting state for each soft-thread and then generates the same first value for each pixel, whereas when the rng keyword is used, the RNG is shared and does not. Since unwrap accepts an rng kwarg it should still behave in a stable fashion given some RNG instance, rather than only if used with the task-local RNG.

@wheeheee
Copy link
Member

I cannot reproduce with the default TaskLocalRNG, MersenneTwister, or StableRNG if I do it like this

julia> measure(s, rng, seed) = (unwrap(s; dims=1:2, rng=rng(seed)) - unwrap(s; dims=1:2, rng=rng(seed))).^2 |> mean |> sqrt
measure (generic function with 1 method)

julia> ps = angle.(rand(ComplexF64, 1000, 1000));

julia> measure(ps, StableRNG, 10) == measure(ps, MersenneTwister, 10) == measure(ps, Random.default_rng, 10) == 0
true

However, there is indeed a difference with

julia> ps = rand(1000, 1000) * 1234;

julia> @test measure(ps, StableRNG, 10) == measure(ps, MersenneTwister, 10) == measure(ps, Random.default_rng, 10) == 0
Test Failed at REPL[11]:1
  Expression: measure(ps, StableRNG, 10) == measure(ps, MersenneTwister, 10) == measure(ps, Random.default_rng, 10) == 0
   Evaluated: 294.78516174681107 == 160.76991914304145 == 177.74442713272344 == 0

ERROR: There was an error during testing

Inspecting the output of different runs of unwrap here, I noticed that they differ by multiples of 2pi. The documentation states that unwrap requires the values of the input array to be in a certain range, which is probably why I get these results. Maybe your input is similarly invalid?

Although yes, this is orthogonal to reproducibility, because even with an invalid input we should maybe get the same invalid output...

@RomanHargrave
Copy link
Author

RomanHargrave commented Dec 29, 2024

The theory about invalid input is interesting - I was initially going to suggest something about the nature of the random data before I got distracted with the potentially wrong conclusion about task-local RNG.

Per the DSP.jl docs,

A common usage for unwrapping across a singleton dimension is for a phase measurement over time, such as when comparing successive frames of a short-time-fourier-transform, as each frame is wrapped to stay within (-pi, pi].

A common usage for unwrapping across multiple dimensions is for a phase measurement of a scene, such as when retrieving the phase information of of an image, as each pixel is wrapped to stay within (-pi, pi].

Indeed, the range of my FT phase angle data is within (-pi, pi]

julia> any(-pi .>= ps .> pi)  
false

Anyways, correctness of inputs aside - yes - the behavior should be consistent between applications and means of seeding.

@wheeheee
Copy link
Member

That inequality looks wrong. Indeed,

julia> f(x) = -pi >= x > pi
f (generic function with 1 method)

julia> @code_llvm debuginfo=:none f(100)
; Function Signature: f(Int64)
; Function Attrs: uwtable
define i8 @julia_f_9255(i64 signext %"x::Int64") #0 {
top:
  ret i8 0
}

Could you try instead

all(-pi .< ps .<= pi) # returns false if there is invalid input

@RomanHargrave
Copy link
Author

RomanHargrave commented Dec 29, 2024

Ah, you are correct. I've been staring at the computer for too long.

So, the minimum of the live phase angle data is -pi in accordance with the range of angle (-pi <= angle() <= pi), which does put it out of domain by a small margin.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants