Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

update-backlog: triage not-not PASSing tests #109

Open
ErichDonGubler opened this issue May 30, 2024 · 2 comments · May be fixed by #111
Open

update-backlog: triage not-not PASSing tests #109

ErichDonGubler opened this issue May 30, 2024 · 2 comments · May be fixed by #111
Assignees
@ErichDonGubler
Labels
A-webgpu-cts enhancement New feature or request

Comments

@ErichDonGubler
Copy link
Collaborator

There are quite a few tests that expect PASS, TIMEOUT, and NOTRUN outcomes (or some subset) at the test and subtest level, but not FAIL or CRASH outcomes. The key observation here is we haven't actually observed these to not PASS; I'll call these "not-not PASSing tests". My hypothesis is that many of these will simply be PASSing when we run those tests in tier 2's broader timeout window. We should triage not-not PASSing tests with a new heuristic for update-backlog.
@ErichDonGubler
Copy link
Collaborator Author

Estimate: S (~1.5 days; ~0.5 days to implement, and 1 to triage).

@ErichDonGubler
Copy link
Collaborator Author

Expected to yield at least a dozen promoted tests.

@ErichDonGubler ErichDonGubler added enhancement New feature or request A-webgpu-cts labels May 30, 2024
@ErichDonGubler ErichDonGubler self-assigned this May 30, 2024
This was referenced May 31, 2024
Open
Merged
@ErichDonGubler ErichDonGubler linked a pull request Jun 6, 2024 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ErichDonGubler ErichDonGubler

Labels
A-webgpu-cts enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(update_backlog): add --preset=promote-not-not-passing erichdongubler-mozilla/moz-webgpu-cts
1 participant
@ErichDonGubler