Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Confusing numbers in histogram #249

Closed
giordano opened this issue Aug 20, 2021 · 3 comments · Fixed by #250
Closed

Confusing numbers in histogram #249

giordano opened this issue Aug 20, 2021 · 3 comments · Fixed by #250

Comments

@giordano
Copy link
Contributor

giordano commented Aug 20, 2021

Trying to reproduce the benchmarks in this discourse post, I've got

ulia> @benchmark $a[$w] .= $b[$w]
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  10.176 μs …  1.150 ms  ┊ GC (min … max): 0.00% … 98.96%
 Time  (median):     10.987 μs              ┊ GC (median):    0.00%
 Time  (mean ± σ):   13.660 μs ± 17.326 μs  ┊ GC (mean ± σ):  3.08% ±  2.71%

  █                                                            
  █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▁
  53.4 μs         Histogram: frequency by time        11.1 μs <

 Memory estimate: 39.33 KiB, allocs estimate: 2.

The mean time is 13.660 μs ± 17.326 μs, I haven't looked at the code but the standard deviation larger than the average time itself looks suspiciously wrong (or maybe large one-sided outliers are screwing it up?). 👈 Yes, I believe the problem is that the distribution isn't normal, there are large outliers only on one side, so the standard deviation is messed up. The issue below still stands though 👇

Also, note that the histogram has 53.4 μs on the left and 11.1 μs on the right: I have no idea how to read this histogram, is it reversed?

@giordano giordano changed the title Confusing numbers in histogram and mean time has standard deviation larger than average time itself Confusing numbers in histogram Aug 20, 2021
@giordano
Copy link
Contributor Author

Other examples of funny numbers in the histogram: https://discourse.julialang.org/t/why-is-minimum-so-much-faster-than-argmin/66814/8

@tecosaur
Copy link
Collaborator

Yea, that's funky. I wonder if it has anything to do with my assumption that times were sorted? In #245 this assumption looks like it could be a bit dodgy.

@vchuravy
Copy link
Member

#240 times are not sorted

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants