Improve GitHub PR comments on flaky tests #2047
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This workflow needs to be in the main repo to test it (so this CI doesn't test this PR). What changes do is basically to download (optionally) flaky test reports (if present). And then use JBang script to create comment. I am using JBang script because parsing of the report is non-trivial. Any future improvements can be driven from the Flaky Run Reporter for both TS and FW.
I have tried it locally and I also wrote automated tests for that JBang script, but I cannot guarantee this will work OOTB.
One thing I am not sure about is uniqueness of the artifact names in case there is matrix within same job with multiple flakes. It's not exactly easy to figure from GH docs so far, so I say we shall see.
Please select the relevant options.
run tests
phrase in comment)Checklist: