GitHub reporter

[hugo-vrijswijk]

GitHub has something called 'check-runs' they allow adding annotations to a 'check'. Checks are for example what GitHub Actions create. These annotations show up at a line of code. A good example is the eslint-action which lints code and displays error messages directly in GitHub. More info about checks can be found here or here. Annotations can be added to a new check created by the action, or an existing one that is currently running like the CI action. It looks a little like this:

We could add something similar to the Stryker Dashboard by adding a GitHub reporter. This reporter could annotate survived mutants at their position in the source code on GitHub so people can view their mutation testing results in GitHub. This would also work very well in combination with a diff-runner (not supported yet by Stryker4s). A score could also be reported at the end

Some things that need discussion:

• What should survived mutants be reported as? I don't think it should be an 'error' because then the user will always have a failed check unless they have a score of 100% which is the wrong message to send IMO
• Should we report other mutant statuses? It might become a little crowded if we report everything
• Another option would be a stryker-report-action that could be run after mutation testing. The action could read the JSON report and send the annotations. Upside is that it would work for all Strykers (or other mutation testing framework). Downside is it would not report the results in realtime, and would require the main CI to be in GitHub actions already (which wouldn't be required if the Stryker4s reporter creates a new check)
• Do we currently have enough information in the dashboard to make this API call
• Easiest way to do it would be when the framework sends the full report. Coolest way would be in a streaming fashion as results come in

[rouke-broersma]

This is something I think would fit better as a feature of the dashboard, where you can configure the dashboard to post the result to github as a status check. That way we only have to implement it once, and adds an extra use case for using the dashboard.

This is also how other services like sonarcloud work.

[cbrunnkvist]

As @Mobrockers points out, if submitting the check instead is a responsibility of the dashboard, then you can avoid having one GitHub API client to this in stryker4s, and another one in stryker-net and so on.

There are a couple of other things to put on the wish list here: reporting changes in score between builds/checks is a major one. Features like that (tracking reports by commit rather then by branch) also fit much better in the dashboard project.

[hugo-vrijswijk]

I agree it does make more sense for this functionality to be in the Stryker dashboard. I'll move this issue there and edit the description

[johanholmerin]

For anyone interested in this, I have made a Github Action that creates annotations from a Stryker report.

[rouke-broersma]

@johanholmerin our json schema is actually not Stryker specific :) We design and publish the schema but iirc other mutation testing tools also implement this schema. So your action is useful not only to Stryker users but also other mutation testing tool users!

[johanholmerin]

I did not know that. I've changed the name to make it clearer that it's not specific to Stryker

[xandervedder]

Do we still want this functionality in the Stryker dashboard? Or does the action mentioned above provide the functionality that was wanted originally?

[rouke-broersma]

At some point I would like to see this integrated into the dashboard similar to for example sonarqube.