Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add benchmarks for Gen.sample #373

Merged
merged 4 commits into from Nov 3, 2021
Merged

Add benchmarks for Gen.sample #373

merged 4 commits into from Nov 3, 2021

Conversation

ghost
Copy link

@ghost ghost commented Sep 23, 2021

This PR puts the examples from #135 into a proper benchmark, in support of #371.

/cc @moodmosaic @TysonMN

@ghost
Copy link
Author

ghost commented Sep 23, 2021

BenchmarkDotNet=v0.13.1, OS=debian 10 (container)
Intel Core i7-9750H CPU 2.60GHz, 8 CPU, 8 logical and 8 physical cores
.NET Core SDK=3.1.413
  [Host]        : .NET Core 3.1.19 (CoreCLR 4.700.21.41101, CoreFX 4.700.21.41603), X64 RyuJIT DEBUG
  .NET Core 3.1 : .NET Core 3.1.19 (CoreCLR 4.700.21.41101, CoreFX 4.700.21.41603), X64 RyuJIT

Job=.NET Core 3.1  Runtime=.NET Core 3.1  
Method Mean Error StdDev
GenSampleLargeHedgehog 130.46 ms 1.178 ms 0.984 ms
GenSampleLargeFsCheck 41.73 ms 0.216 ms 0.191 ms

@ghost
Copy link
Author

ghost commented Sep 23, 2021

Also, as a fun side note I removed bigint again (again, naively just replacing bigint with int64) and got the following:

BenchmarkDotNet=v0.13.1, OS=debian 10 (container)
Intel Core i7-9750H CPU 2.60GHz, 8 CPU, 8 logical and 8 physical cores
.NET Core SDK=3.1.413
  [Host]        : .NET Core 3.1.19 (CoreCLR 4.700.21.41101, CoreFX 4.700.21.41603), X64 RyuJIT DEBUG
  .NET Core 3.1 : .NET Core 3.1.19 (CoreCLR 4.700.21.41101, CoreFX 4.700.21.41603), X64 RyuJIT

Job=.NET Core 3.1  Runtime=.NET Core 3.1  
Method Mean Error StdDev
GenSampleLargeHedgehog 52.42 ms 0.695 ms 0.616 ms
GenSampleLargeFsCheck 42.26 ms 0.352 ms 0.330 ms

@ghost ghost added this to the 0.12.0 milestone Sep 23, 2021
@moodmosaic
Copy link
Member

I need to refresh my memory on the choice for bigint, perhaps because of the stuff in Numeric.fs 🤔

@TysonMN
Copy link
Member

TysonMN commented Sep 24, 2021

Add Params like in this file. I am thinking powers of 10 would be good choices.

And this slightly contradicts my other comment that said to put the quantity to be sampled in the method name. With this change, don't put the quantity in the name.

@ghost
Copy link
Author

ghost commented Sep 24, 2021

I need to refresh my memory on the choice for bigint, perhaps because of the stuff in Numeric.fs 🤔

I think it was out of concern for generating min/max ranges for uint64/int64? I remember seeing a comment from you on some issue or PR about it. Either way, I think we can safely replace it 🤷🏼‍♂️

@ghost
Copy link
Author

ghost commented Sep 24, 2021

Add Params like in this file. I am thinking powers of 10 would be good choices.

And this slightly contradicts my other comment that said to put the quantity to be sampled in the method name. With this change, don't put the quantity in the name.

BenchmarkDotNet=v0.13.1, OS=debian 10 (container)
Intel Core i7-9750H CPU 2.60GHz, 8 CPU, 8 logical and 8 physical cores
.NET Core SDK=3.1.413
  [Host]        : .NET Core 3.1.19 (CoreCLR 4.700.21.41101, CoreFX 4.700.21.41603), X64 RyuJIT DEBUG
  .NET Core 3.1 : .NET Core 3.1.19 (CoreCLR 4.700.21.41101, CoreFX 4.700.21.41603), X64 RyuJIT

Job=.NET Core 3.1  Runtime=.NET Core 3.1
Method N Mean Error StdDev Median
HedgehogGenSampleInt32 100 139.42 μs 1.612 μs 1.508 μs 139.31 μs
FsCheckGenSampleInt32 100 23.39 μs 0.065 μs 0.054 μs 23.38 μs
HedgehogGenSampleInt32 1000 1,313.18 μs 17.493 μs 16.363 μs 1,304.07 μs
FsCheckGenSampleInt32 1000 169.81 μs 3.191 μs 4.260 μs 170.33 μs
HedgehogGenSampleInt32 10000 13,045.48 μs 185.140 μs 173.180 μs 13,081.98 μs
FsCheckGenSampleInt32 10000 1,878.52 μs 31.893 μs 29.833 μs 1,886.14 μs
HedgehogGenSampleInt32 100000 130,364.13 μs 1,713.186 μs 1,602.515 μs 130,159.68 μs
FsCheckGenSampleInt32 100000 27,397.14 μs 542.102 μs 1,309.237 μs 26,793.29 μs

For reference, here's the same benchmarks without bigint:

BenchmarkDotNet=v0.13.1, OS=debian 10 (container)
Intel Core i7-9750H CPU 2.60GHz, 8 CPU, 8 logical and 8 physical cores
.NET Core SDK=3.1.413
  [Host]        : .NET Core 3.1.19 (CoreCLR 4.700.21.41101, CoreFX 4.700.21.41603), X64 RyuJIT DEBUG
  .NET Core 3.1 : .NET Core 3.1.19 (CoreCLR 4.700.21.41101, CoreFX 4.700.21.41603), X64 RyuJIT

Job=.NET Core 3.1  Runtime=.NET Core 3.1  
Method N Mean Error StdDev
HedgehogGenSampleInt32 100 58.38 μs 1.010 μs 1.349 μs
FsCheckGenSampleInt32 100 23.51 μs 0.071 μs 0.066 μs
HedgehogGenSampleInt32 1000 500.94 μs 2.716 μs 2.407 μs
FsCheckGenSampleInt32 1000 167.69 μs 2.665 μs 2.362 μs
HedgehogGenSampleInt32 10000 5,187.75 μs 27.047 μs 22.586 μs
FsCheckGenSampleInt32 10000 1,864.81 μs 35.988 μs 36.957 μs
HedgehogGenSampleInt32 100000 50,216.08 μs 913.421 μs 1,551.057 μs
FsCheckGenSampleInt32 100000 26,470.79 μs 357.376 μs 334.289 μs

image

There is the data graphed, green is Hedgehog+BigInt, Blue is Hedgehog+Int64, Black is FsCheck.

@TysonMN
Copy link
Member

TysonMN commented Sep 25, 2021

This looks great. Excellent work @adam-becker!

Copy link
Member

@TysonMN TysonMN left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this PR is ready to merge

@TysonMN
Copy link
Member

TysonMN commented Oct 15, 2021

@adam-becker, do you think this PR is ready to merge?

@dharmaturtle
Copy link
Member

I don't have any experience with BenchmarkDotNet - is it possible to use it to write a test that will fail if, say, HedgehogGenSampleInt32 N=100 takes 1 second? Not a blocker to merging - I'm just wondering.

@TysonMN
Copy link
Member

TysonMN commented Oct 25, 2021

I don't think so. The point of a BenchmarkDotNet test is to look at the report it generates.

@TysonMN TysonMN merged commit b737446 into hedgehogqa:master Nov 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants