Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support global deterministic seed #450

Open
cmeeren opened this issue Oct 6, 2023 · 0 comments
Open

Support global deterministic seed #450

cmeeren opened this issue Oct 6, 2023 · 0 comments

Comments

@cmeeren
Copy link
Contributor

cmeeren commented Oct 6, 2023

Not sure if this is feasible, but I'll try:

I like deterministic builds. Being able to easily re-build and re-deploy a previous build has saved me many times. (Let's keep aside the roll-forward vs. roll-back debate. It's irrelevant anyway; the below also applies to a roll-forward.)

I have at times been thwarted by Hedgehog properties failing on CI builds where it did not fail locally, and where re-running makes it pass. Or vice versa; failing locally, but unexpectedly passing on CI. The reason is that a rare combination of generated values surfaces a bug (most often this is in the test/generator code, though occasionally it is in the production code). This combination does not occur on every run.

I would like to bring determinism back to my test suite, by having Hedgehog use the same seed every time, so to speak. So that, sure, when it runs a property 100 times, the values are seemingly random, but subsequent runs of the test produces the same values for each respective iteration. I want to be able to run a test as many times as I want, and never get different results, for the aforementioned reasons.

The only conceptual drawback I see to this is that making the seed random on every run increases the chance of catching a bug. But personally, that is not a way I want to catch a bug. It indicates a poorly written property that does not sufficiently try to reproduce relevant edge cases. I value deterministic builds more. If I want increased confidence in a property, I just increase the property's test count.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant