Add update-backlog
subcommand with first promote-perma-passing
preset
#83
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We at Mozilla recently added the
implementation-status: backlog
property to all test cases (see upstream docs on metadata), so we can create a workflow for incrementally making WebGPU CTS visible to Firefox's CI sheriffing team. The idea is to incrementally remove it as we determine contradictory test results are worth filing bugs for. We now useimplementation-status
for both tier 2 and tier 3 of Firefox CI. I'm currently calling the migration of a test from a less stable tier to a more stable tier by removingimplementation-status: backlog
a "promotion".We do experiments in promoting tests according to heuristics like "promote permanently
PASS
ing tests" (already implemented here), "promote tests that aren't observed toFAIL
orCRASH
" (to be implemented in #109), with more to come. The workflow then becomes:wptreport.json
files.update-expected
as desired forbacklog
ged tests, and create a commit based on changes.update-backlog
to tentatively promote tests. Commit and submit to CI as an experiment.We consider using
implementation-status: backlog
to be valuable because:wptrunner
has CLI support for filtering tests by implementation status, so the lift to adjust our CI was light.expected
; sometimes, we want to model that a test should pass, but not block CI on it yet; sometimes, we want a test to be expected toFAIL
, and block CI when it startsPASS
ing. Concretely, the former case came up recently with bug 1897131.Original OP
I'm currently planning on separating the corpus of WebGPU CTS test runs in Firefox CI into testing based on the
implementation-status
(see bug 1873687). I'm using this PR as a way to explore automatically changing theimplementation-status
.