Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Measure code overhead depending on the environment the tests run in #520

Merged
merged 1 commit into from
Aug 29, 2020

Conversation

nolar
Copy link
Owner

@nolar nolar commented Aug 29, 2020

What do these changes do?

Fix the CI pipelines which fail often due to unpredictable timing of time-based tests.

Description

Some tests rely on the timing of the code-under-tests: e.g. the watcher/worker/queueing/batching of watch-events into the processing callbacks.

In the absence of freezegun-like time-shifting for asyncio, the tests are designed by directly executing the code with small timeouts, and measuring the actual time taken. The small timeouts are needed to keep the whole test suite fast.

As a result of such design, the actual duration of tests is often close to or significantly influenced by the code overhead, which in turn varies from environment to environment, from run to run — thus failing the assertions.

To address this properly, the timeouts should be made high, and the time movement should be simulated instead. We cannot do this now.

To address this sufficiently, the code overhead can be measured in the environment where the tests are executed, and taken into account instead of the hard-coded empirically-guessed constant.

This PR adds such measurements on a per-test basis as a fixture (the measurement itself takes a fraction of a second) — in the same setup as the tests use. It supposedly solves the problem with CI tests failing due to timing and being unable to re-run properly.

Issues/PRs

Issues: #212 #338

Type of changes

  • Bug fix (non-breaking change which fixes an issue)
  • Mostly CI/CD automation, contribution experience

Checklist

  • The code addresses only the mentioned problem, and this problem only
  • I think the code is well written
  • Unit tests for the changes exist
  • Documentation reflects the changes
  • If you provide code modification, please add yourself to CONTRIBUTORS.txt

@nolar nolar added bug Something isn't working automation CI/CD: testing, linting, releasing automatically labels Aug 29, 2020
@nolar nolar force-pushed the aiohttp-timed-tests branch from 1970895 to bbffbcf Compare August 29, 2020 17:27
@nolar nolar merged commit 65a33e3 into master Aug 29, 2020
@nolar nolar deleted the aiohttp-timed-tests branch August 29, 2020 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automation CI/CD: testing, linting, releasing automatically bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant