Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add update-backlog subcommand with first promote-perma-passing preset #83

Merged
merged 2 commits into from
Jun 6, 2024

Conversation

ErichDonGubler
Copy link
Collaborator

@ErichDonGubler ErichDonGubler commented Apr 3, 2024

We at Mozilla recently added the implementation-status: backlog property to all test cases (see upstream docs on metadata), so we can create a workflow for incrementally making WebGPU CTS visible to Firefox's CI sheriffing team. The idea is to incrementally remove it as we determine contradictory test results are worth filing bugs for. We now use implementation-status for both tier 2 and tier 3 of Firefox CI. I'm currently calling the migration of a test from a less stable tier to a more stable tier by removing implementation-status: backlog a "promotion".

We do experiments in promoting tests according to heuristics like "promote permanently PASSing tests" (already implemented here), "promote tests that aren't observed to FAIL or CRASH" (to be implemented in #109), with more to come. The workflow then becomes:

  1. Run tests, and gather wptreport.json files.
  2. Run update-expected as desired for backlogged tests, and create a commit based on changes.
  3. Run update-backlog to tentatively promote tests. Commit and submit to CI as an experiment.
  4. Check tentatively promoted tests are successful in the above experiment. Where they are not, demote them, preferably with bugs in Bugzilla.

We consider using implementation-status: backlog to be valuable because:

  • wptrunner has CLI support for filtering tests by implementation status, so the lift to adjust our CI was light.
  • It's a clear signal that a test is not yet considered valuable to run as a blocker in CI yet.
  • It's orthogonal to expected; sometimes, we want to model that a test should pass, but not block CI on it yet; sometimes, we want a test to be expected to FAIL, and block CI when it starts PASSing. Concretely, the former case came up recently with bug 1897131.

Original OP

I'm currently planning on separating the corpus of WebGPU CTS test runs in Firefox CI into testing based on the implementation-status (see bug 1873687). I'm using this PR as a way to explore automatically changing the implementation-status.

@ErichDonGubler ErichDonGubler force-pushed the update-backlog branch 3 times, most recently from ef6b6ee to a856ba3 Compare April 18, 2024 18:50
@ErichDonGubler ErichDonGubler force-pushed the update-backlog branch 3 times, most recently from 9fc58ea to c91b937 Compare May 7, 2024 21:59
@ErichDonGubler

This comment was marked as resolved.

@ErichDonGubler ErichDonGubler changed the title Add update-backlog subcommand Add update-backlog subcommand with first promote-perma-passing preset Jun 6, 2024
@ErichDonGubler ErichDonGubler marked this pull request as ready for review June 6, 2024 19:25
@ErichDonGubler ErichDonGubler enabled auto-merge June 6, 2024 19:26
@ErichDonGubler ErichDonGubler merged commit 887db4d into main Jun 6, 2024
32 checks passed
@ErichDonGubler ErichDonGubler deleted the update-backlog branch June 6, 2024 19:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant