-
Notifications
You must be signed in to change notification settings - Fork 667
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mutation testing - pr differences #4178
Mutation testing - pr differences #4178
Conversation
- dockerfile and shell script for specific packages - ci.yml for diff on packages on PR
Co-authored-by: jbencin <[email protected]>
- runs for modified files & created files - has to be run before committing the changes ``` cd mutation-testing/scripts sh git-diff.sh ```
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #4178 +/- ##
===========================================
+ Coverage 82.43% 82.59% +0.16%
===========================================
Files 402 401 -1
Lines 291084 288404 -2680
===========================================
- Hits 239951 238205 -1746
+ Misses 51133 50199 -934 ☔ View full report in Codecov by Sentry. |
moved to draft so @ASuciuX et al can work through some issues discovered in initial tests |
The |
Cases: - if >= 16 mutants on big packages => run big packages using 8 shards - else run big packages without shards - if >= 16 mutants on big packages and >= 120 mutants on small packages => run small packages using 4 shards - else if < 16 mutants on big packages and >= 80 mutants on small packages => run small packages using 4 shards - else run small packages without shards
Overall Flow Diagram #4210 (comment) |
I'm also considering that you may want to run this workflow only after an approval - otherwise, it will currently trigger on any commit to the branch being PR'ed. |
The goal here was to run the mutants before the PR's approval, since the workflow is a reference to which of the added/modified functions fail when running mutants, so the developers working on the PR can check and fix them and the others can approve the PR based on the fact that there are no mutants unchecked. |
Technically only a job out of the 4 calling the composite action could be moved - one without strategy matrix - but it would require additional checks in the composite. The other 2 jobs have different matrix dimensions, so there's really no way to get rid of them, since at least 1 job would run without matrix, one would run with 4 and the other with 8 shards. The job with the verification that's done with bash scripts before them is made in order to know which one of the jobs to run, and the only thing that is left to be moved to composite is the output job, which would decrease the lines count by ~35%, but we would still have over 200. If that's the case, do you think I should still move the output job to a composite action, or leave it here? |
@ASuciuX it's entirely doable - there is a LOT of shell in this workflow. moving that to a different repo would make fixing/updating it much easier. looking it through, i don't think it's an easy task (mainly due to the checks/outputs you have) but i think it's worthwhile nonetheless for the reason i stated above. it would require some care in the composite action (mainly around inputs/outputs), but again - i feel it's worth the effort and will make maintaining this workflow much easier. |
- add required input fields to the output job
this is interesting @ASuciuX : https://github.com/stacks-network/stacks-core/actions/runs/7547829177/job/20548598881?pr=4178#step:2:1323 even more peculiar is that it tried to download a musl package to satisfy:
|
I've reviewed past worflow executions and noticed that this issue has been consistently present in all of them. I believe it will work alright, but i could change it to https://github.com/cargo-bins/cargo-binstall?tab=readme-ov-file#in-github-actions. The interesting part is that binstall and also the mutants docs recommend using the one I've added. |
my concern is mostly that the workflow failed and nothing else ran as a result:
do you know why that error was triggered? |
@wileyj thanks for pointing it. just fixed it stacks-network/actions#14 it was related to: renaming |
@ASuciuX the last question i have here is timing. if i understand correctly, the idea is that this workflow will run on every PR. can you share some timings that we can expect based on different types of PR's? |
I've identified and resolved an issue with the git diff command used in our workflow, which you can review here. The workflow run details are available here. Previously, the command was erroneously capturing changes not only from the source branch but also from the base branch, starting from the initiation of the PR up to the present. This resulted in an extended runtime of about 3 hours for the mutants process, encompassing all differences on the
An enhancement that could significantly optimize our runtime involves dynamically terminating mutant processes based on their initial test durations. For instance, if the initial test for stacks-node takes X minutes, we could set a threshold to automatically stop the process after 1.5X minutes and call that a For context, in previous runs:
With the fix applied, the workflow should now correctly process only the relevant changes, leading to more efficient runtimes. |
On top of the above specified run times, the integration of |
this is great!
to the file ./docs/ci-release.md? ideally under a section regarding the mutant workflow |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i think one this documentation comment is addressed, everything else looks good
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shipit
Applicable issues
This has to be merged before in order to have the references right. Please review this as well
Additional info (benefits, drawbacks, caveats)
Solutions and Recommended Usage
Cargo-mutant does an incremental build for each mutation, that’s why this solution comes with the implementation of tracking the output and only updating it with the differences, to run 10-20, maybe a few 100s mutants instead of 20,000.
CI YML File: Are linked with the actions in the actions' repo stacks-network/actions#3.
Should be executed with every PR, as it tests only the updated or newly added functions in the PR's commits.
How to read the status from the
pr-differences
workflowIf it is not failing, it means all the functions are tested properly.
If it is failing it will display what type of tests are the problem: missing tests, timeout tests, or unviable tests. For each section, it will specify the functions, the files they are in and their line numbers.
Handling Exit Codes (docs)
Mutation testing produces exit codes post-completion.
In the
pr-differences
workflow they are caught and specified with extra details about what should be done based on them.