You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I like deterministic builds. Being able to easily re-build and re-deploy a previous build has saved me many times. (Let's keep aside the roll-forward vs. roll-back debate. It's irrelevant anyway; the below also applies to a roll-forward.)
I have at times been thwarted by Hedgehog properties failing on CI builds where it did not fail locally, and where re-running makes it pass. Or vice versa; failing locally, but unexpectedly passing on CI. The reason is that a rare combination of generated values surfaces a bug (most often this is in the test/generator code, though occasionally it is in the production code). This combination does not occur on every run.
I would like to bring determinism back to my test suite, by having Hedgehog use the same seed every time, so to speak. So that, sure, when it runs a property 100 times, the values are seemingly random, but subsequent runs of the test produces the same values for each respective iteration. I want to be able to run a test as many times as I want, and never get different results, for the aforementioned reasons.
The only conceptual drawback I see to this is that making the seed random on every run increases the chance of catching a bug. But personally, that is not a way I want to catch a bug. It indicates a poorly written property that does not sufficiently try to reproduce relevant edge cases. I value deterministic builds more. If I want increased confidence in a property, I just increase the property's test count.
The text was updated successfully, but these errors were encountered:
Not sure if this is feasible, but I'll try:
I like deterministic builds. Being able to easily re-build and re-deploy a previous build has saved me many times. (Let's keep aside the roll-forward vs. roll-back debate. It's irrelevant anyway; the below also applies to a roll-forward.)
I have at times been thwarted by Hedgehog properties failing on CI builds where it did not fail locally, and where re-running makes it pass. Or vice versa; failing locally, but unexpectedly passing on CI. The reason is that a rare combination of generated values surfaces a bug (most often this is in the test/generator code, though occasionally it is in the production code). This combination does not occur on every run.
I would like to bring determinism back to my test suite, by having Hedgehog use the same seed every time, so to speak. So that, sure, when it runs a property 100 times, the values are seemingly random, but subsequent runs of the test produces the same values for each respective iteration. I want to be able to run a test as many times as I want, and never get different results, for the aforementioned reasons.
The only conceptual drawback I see to this is that making the seed random on every run increases the chance of catching a bug. But personally, that is not a way I want to catch a bug. It indicates a poorly written property that does not sufficiently try to reproduce relevant edge cases. I value deterministic builds more. If I want increased confidence in a property, I just increase the property's test count.
The text was updated successfully, but these errors were encountered: