Coverage with bisect_ppx

[vch9]

As we are adding tests thanks to @jmid, we could add bisect_ppx to produce a coverage report.

[jmid]

Getting an idea of our coverage would be nice!

A few thoughts:

• Have you tried running Coverage report with bisect_ppx #206 to see how bad it is? 😅
• I was wondering how we could bring it actively into play - rather than just carry the instrumentation lines around:
  • a CI-badge reporting the percentage in the README?
  • a CI alarm that triggers if a PR decreases coverage? (could be useful to catch PRs without tests - but potentially annoying...)
  • ...
• Finally, if we're going to measure how well our testsuite is doing it would be nice to rebase and squash the test PRs from the fall (sans the shrink logging for a start). I'll see if I can find the time...

[vch9]

Have you tried running Coverage report with bisect_ppx #206 to see how bad it is? sweat_smile

It's not that bad! A checkout to a commit 6 months ago would show a way funnier percentage :).

✦8 Ψ bisect-ppx-report summary --per-file                                 
  0.00 %      0/0      src/QCheck_runner.ml
 78.57 %     22/28     src/alcotest/QCheck_alcotest.ml
 63.93 %    927/1450   src/core/QCheck.ml
 67.21 %    859/1278   src/core/QCheck2.ml
 12.50 %     20/160    src/ounit/QCheck_ounit.ml
100.00 %      2/2      src/ppx_deriving_qcheck/QCheck_generators.ml
 87.50 %      7/8      src/ppx_deriving_qcheck/args.ml
 88.89 %      8/9      src/ppx_deriving_qcheck/attributes.ml
 96.33 %    236/245    src/ppx_deriving_qcheck/ppx_deriving_qcheck.ml
100.00 %     53/53     src/ppx_deriving_qcheck/tuple.ml
 64.87 %    181/279    src/runner/QCheck_base_runner.ml
 65.92 %   2315/3512   Project coverage

I was wondering how we could bring it actively into play - rather than just carry the instrumentation lines around:

We can have a badge in the README yes, we also can have a diff percentage in each PR. I also think an alarm when its decreasing will be annoying, the only alarm I'd think would be useful is to detect loss covered lines, but that's practically impossible to do.