Tracking issue: moz-webgpu-cts update-expected options

[ErichDonGubler]

Tracked issues

• MVP: Introduce moz-webgpu-cts process-reports #13
• --preset=merge for moz-webgpu-cts process-reports #9
• Introduce --on-missing-from-report=… option for update-expected #25
• Manual resolution: Add an interactive prompt-driven flow for moz-webgpu-cts update-expected … #20
• WIP: permas-only mode for process-reports a la mach wpt-update with no further options #27; TODO: fill out new issue and table below
  • Add --reorder-intermittents option to moz-webgpu-cts process-reports #28

Preset modes

Name	--on-conflict=…	--on-missing-from-report=-…	--reorder-intermittents=…	--explicit-pass=…
webgpu-new-fx-changes (TODO: file issue)	change-to-reported	keep	false	true
gather-intermittents (TODO: file issue)	keep-both	keep	false	true
wpt-update (TODO: file issue)	change-to-reported	keep	true	false
regen-from-reports (TODO: file issue)	change-to-reported	remove	true	remove

Known workflows

How to handle intermittents	Steps	Places where this workflow is used
Handle intermittents as backlog encoded in metadata	Changeset author(s) process an initial set of reports with --preset=webgpu-new-fx-changes once. Optionally, changeset author(s) process more reports with --preset-gather-intermittents until intermittent CI failures are acceptable. Process intermittent failure reports with --preset-webgpu-gather-intermittents as intermittents are discovered.	WebGPU team
Handle intermittents as typical items of work	Changeset author(s) process an initial set of reports with --preset-wpt-update once. Optionally, changeset author(s) process more reports with --preset=gather-intermittents to triage (but not necessarily commit to metadata) intermittents that have been potentially introduced by their change. Over time, team members are pinged by a separate triaging entity (Mozilla's automation and/or sheriffing, for instance) when intermittents are discovered in CI.	Most of Mozilla?

Transition table

Symbol	Meaning
👍	No changes should be made.
💥	Conflict detected. Consult --on-conflict=… for resolution: change-to-reported will overwrite current metadata with the aggregation of all reported outcomes. keep-both will merge all reported outcomes to current metadata outcome(s), changing to an intermittent set if necessary. Implicit PASSes in metadata are promoted to explicit PASSes. Consult --reorder-intermittents=… for ordering of new metadata outcomes: true: Order the entire set of outcomes by observed frequency. If not observed in reports, current metadata outcomes should be assigned a frequency a 0 in this sorting operation, with existing order as a tie-breaker. false: Append outcome(s) not present in current metadata in order of observed frequency.
❓	An explicit permanent PASS will be removed from metadata, unless --explicit-pass-in-meta=keep is specified.
🫥	Test(s) and their subtest(s) will be removed from metadata because they were not encountered in test execution, unless --on-missing-from-report=keep is specified.
❗	Though all reported outcomes are expected in current metadata, they will be recorded into metadata by order of observed frequency unless --reorder-intermittents=false. If not observed in reports, current metadata outcomes should be assigned a frequency a 0 in this sorting operation, with existing order as a tie-breaker.

📁Outcomes expected in metadata	🔍Outcomes reported by test executions	Cases to consider
Missing (implicit permanent PASS)	Exact match (permanent PASS)	👍
Missing (implicit permanent PASS)	Missing
Missing (implicit permanent PASS)	Permanent, does not match (non-PASS)	💥
Missing (implicit permanent PASS)	Intermittent, includes match
Missing (implicit permanent PASS)	Intermittent, does not include match
Permanent explicit PASS	Exact match	❓
Permanent explicit PASS	Missing	🫥
Permanent explicit PASS	Permanent, does not match	💥
Permanent explicit PASS	Intermittent, includes match
Permanent explicit PASS	Intermittent, does not include match
Permanent non-PASS or intermittent	Exact match	👍
Permanent non-PASS or intermittent	Missing	🫥
Permanent	Permanent, does not match	💥
Permanent	Intermittent, includes match
Permanent	Intermittent, does not include match
Intermittent	Exact match	❗
Intermittent	Missing	🫥
Intermittent	Permanent, contained in current set	💥
Intermittent	Permanent, not contained in current set
Intermittent	Intermittent, partial intersection
Intermittent	Intermittent, disjoint

[ErichDonGubler]

Only #20 remains of the issues in this list. I don't think a tracking issue carries its weight any more. Closing.