fix: update isolation as new fixtures of certain scopes arrive

[antazoey]

What I did

Make isolation "work" on session, package, module, and class scopes.
Basically, the bug was that as new fixtures of certain scopes come in after a snapshot was taken in that scope, they wouldn't really persist as the scope below left, like if the module changes or something

fixes: #2262
fixes: #1287

How I did it

Awful. Basically the only way to fix this is to rebase the blockchain as new upper scopes up.
A new session comes in later, we have to undo to the module (or closest lowest) snapshot invalidate those fixture, inject those fixtures in the correct order, and re-set all the snapshots.

How to verify it

Can technically uses scopes / parametrized fixtures in any way now... and it should also "work"... it is just is slow if using higher scoped fixtures later on in the test suite, can be really bad actually. I definitely don't recommend using parametrized fixtures unless I can figure out a more optimal way with those

Checklist

• All changes are completed
• New test cases have been added
• Documentation has been updated

[antazoey]

suspecting something with the runner now...

[antazoey]

• Ensure isolation fixtures ran last (this has to be the problem!)

[antazoey]

Update, mentioned on Telegram but I am pretty sure I actually found the bug; I am still working on the full fix this PR is not ready for testing yet). Standby for updates.

[antazoey]

If you want to see the most magical part(s), look at the changes made to the runner.py module.

[antazoey]

This is "working" but may have performance issues if users don't use autouse=True when they should or use parametrized session-scoped fixtures.

[antazoey]

• Either do or consider only reinitializing the autouse=True fixtures (rest should still be invalidated but maybe dont replay them
  What I did: no longer auto reinclude anything. autouse=True fixtures will run again anyway, and the workaround for if you need a fixture to for sure exist, make sure it is used somewhere, even if by another fixture. it actually ain't that hard.

• Ignore fixtures that don't change chain head from invalidation consideration
  Questions: How does one know if a fixture has caused the chain height to increase? There is a way to know if an entire has or if a test has, but there doesn't seem to be a way to know if an individual fixture has, that I can think of.

• For parametrized fixtures, (attempt) to delay the invalidation until the last iteration runs (this will make using them acceptable imo)
  Notes: This seems to work and makes parametrized fixtures ok to use in my book, so yay.

[fubuloubu]

Questions: How does one know if a fixture has caused the chain height to increase? There is a way to know if an entire has or if a test has, but there doesn't seem to be a way to know if an individual fixture has, that I can think of.

Can we store extra info in the cache? Such as block height of last access? If auto-mining is used, that would roughly correspond to 1 block = 1 transaction, so this is a good approximation (If auto-mining is turned off, this would not work)

[antazoey]

Can we store extra info in the cache? Such as block height of last access? If auto-mining is used, that would roughly correspond to 1 block = 1 transaction, so this is a good approximation (If auto-mining is turned off, this would not work)

Yes, this is how we would do it. I just can't figure out how to know which fixture is responsible. It seems really complicated. For example, if you were using multiple module-scoped fixtures at the same time and some caused height increase and some didn't, the chain height would be higher the next usage for all of them, so how does one know which fixture caused the bump?

Edit: Going to see if there a hook for when fixtures run, that may work... liike pytest_fixture_post_finalizer

[antazoey]

• fix parametrized test from apepay
  seems related to parametrized tests, maybe the param treated as a fixture or something

Fixed by 4beeaf3

[antazoey]

• Fix create_stream min time goes to zero issues that has arrived in ApePay

[antazoey]

• Fix create_stream min time goes to zero issues that has arrived in ApePay

Notes:

• Figured out there is a problem when using multiple parametrized fixtures at once.

[antazoey]

• Only trigger invalidation on stateful fixtures of higher scope
• Better handling of parametrized fixtures - right now, always treats them as "new", meaning it is wayy overly invalidating...

[antazoey]

So before I merge this, I am going to add a few more projects to the integration tester thing.
I want to really make sure I am not wrecking all of tests by doing this.

[antazoey]

ApePay

this branch
32 passed in 102.98s (0:01:42)

main
32 passed in 106.16s (0:01:46)

[antazoey]

Once I get #2327 to where I like, I want to also compare running Ape's tests w/ and w/o this feature.