Measure code overhead depending on the environment the tests run in #520
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What do these changes do?
Fix the CI pipelines which fail often due to unpredictable timing of time-based tests.
Description
Some tests rely on the timing of the code-under-tests: e.g. the watcher/worker/queueing/batching of watch-events into the processing callbacks.
In the absence of freezegun-like time-shifting for asyncio, the tests are designed by directly executing the code with small timeouts, and measuring the actual time taken. The small timeouts are needed to keep the whole test suite fast.
As a result of such design, the actual duration of tests is often close to or significantly influenced by the code overhead, which in turn varies from environment to environment, from run to run — thus failing the assertions.
To address this properly, the timeouts should be made high, and the time movement should be simulated instead. We cannot do this now.
To address this sufficiently, the code overhead can be measured in the environment where the tests are executed, and taken into account instead of the hard-coded empirically-guessed constant.
This PR adds such measurements on a per-test basis as a fixture (the measurement itself takes a fraction of a second) — in the same setup as the tests use. It supposedly solves the problem with CI tests failing due to timing and being unable to re-run properly.
Issues/PRs
Type of changes
Checklist
CONTRIBUTORS.txt