Investigate possible nondeterminism in our test suite

[blishko]

I have observed some nondeterministic behaviour with some of our tests, both on my laptop and in CI
From time to time, one test fails (but I don't know if it is the same one every time or not), or some tests take much longer than usual.

[msooseth]

This is due to the fuzz tests I think? It uses a different seed each time for e.g. the simplifier tests that auto generate Expr-s randomly. So that’s 100% expected. Other variations are not really. Although SMT solvers could be nondeterministic, but I hope they aren’t…

…

On Thu 12. Sep 2024 at 09:39, Martin Blicha ***@***.***> wrote: I have observed some nondeterministic behaviour with some of our tests, both on my laptop and in CI From time to time, one test fails (but I don't know if it is the same one every time or not), or some tests take much longer than usual. — Reply to this email directly, view it on GitHub <#553>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXYLY5A4UUXJMH6BALCG6E3ZWFALLAVCNFSM6AAAAABOCRUJP6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGUZDCNRQGEYTANY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[blishko]

I am pretty sure I have seen nondeterministic behaviour in other tests.

[msooseth]

Ooops :S Can you maybe try to track them down? Would be good to know…

…

On Thu 12. Sep 2024 at 09:58, Martin Blicha ***@***.***> wrote: I am pretty sure I have seen nondeterministic behaviour in other tests. — Reply to this email directly, view it on GitHub <#553 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXYLY5CXSFGY5BGZJRTSZQLZWFCQRAVCNFSM6AAAAABOCRUJP6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNBVGUZTCNJWHA> . You are receiving this because you commented.Message ID: ***@***.***>

[blishko]

That's why I created the issue :)
I am just not sure when I will get to that.

[msooseth]

I think what happened is that the abi-encoder fuzzer can sometimes generate a pretty complicated calldata that takes forever to encode. I definitely observed this and it was pretty frustrating. I have opened #554.

I have also labelled this as enhancement, because it doesn't actually affect any user, it's "only" an enhancement for us :)

[msooseth]

So the new additions to #555 could be the trick. There was not only nondeterminsim but a potential bug when running multi-threaded thanks to some monadic magic + concurrently.

[blishko]

I think you were right that the behaviour I was observing were the property-based tests that generate different expressions every time.