Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve GitHub PR comments on flaky tests #2047

Conversation

michalvavrik
Copy link
Member

@michalvavrik michalvavrik commented Sep 25, 2024

Summary

This workflow needs to be in the main repo to test it (so this CI doesn't test this PR). What changes do is basically to download (optionally) flaky test reports (if present). And then use JBang script to create comment. I am using JBang script because parsing of the report is non-trivial. Any future improvements can be driven from the Flaky Run Reporter for both TS and FW.

I have tried it locally and I also wrote automated tests for that JBang script, but I cannot guarantee this will work OOTB.

One thing I am not sure about is uniqueness of the artifact names in case there is matrix within same job with multiple flakes. It's not exactly easy to figure from GH docs so far, so I say we shall see.

Please select the relevant options.

  • Bug fix (non-breaking change which fixes an issue)
  • Dependency update
  • Refactoring
  • Backport
  • New scenario (non-breaking change which adds functionality)
  • This change requires a documentation update
  • This change requires execution against OCP (use run tests phrase in comment)

Checklist:

  • Methods and classes used in PR scenarios are meaningful
  • Commits are well encapsulated and follow the best practices

@mjurc
Copy link
Member

mjurc commented Sep 25, 2024

LGTM, thanks

@michalvavrik
Copy link
Member Author

Only Picocli native failures, but that's expected for now. Thanks

@michalvavrik michalvavrik merged commit 4ab967e into quarkus-qe:main Sep 25, 2024
10 of 14 checks passed
@michalvavrik michalvavrik deleted the feature/improve-flaky-report-comment branch September 25, 2024 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants