EPIC: Preventing and preparing for non-determinism between app versions

[evan-forbes]

Currently, we have minor version bump tests in knuu. This test is great because it performs a simple test for non-determinism between minor versions. However, we likely need to either expand on this test or create new tests that expand on what portions of the state-machine and block validity rules are covered. While not required, ideally we would be able to run some tests upon each push, or at least after being merged to main.

• Non-determinism unit test #2414
• Extend consistent appHash test to include all state machine messages #3540

Unless we are 100% confident that the tests cover every case, then we also need to be prepared to debug non-determinism issues.

• Log module hashes if an app hash mismatch is encountered cosmos-sdk#397
• Dump state and block data upon app hash or last results mismatch and create script to replay cosmos-sdk#398

[evan-forbes]

closing as completed