More Flexible Ratchets - customer request

[vlad-ko]

What product do you want to improve?
Codecov target setting logic for code coverage

Is your feature request related to a problem? Please describe.
Codecov allows users to configure expected coverage for their project using a target specification in codecov.yml – this field takes on either a percentage, or the keyword auto. If a percentage is specified, then the total coverage for the project should not drop below that percentage; if the keyword auto is specified, coverage should never decrease, but no specific threshold of required coverage is enforced.

Most users don’t mess with coverage settings: these are set once, when the project onboards to codecov (often from a template that users don’t normally edit); and are generally not changed after. As such, we’d like to offer teams a broadly applicable configuration that applies to repos in various stages of test maturity and continues to work reasonably well as they grow and mature. Individual orgs within the company may also have specific quality targets/metrics they’re trying to hit, and enforcing consistent coverage config is a tool they would like to use for this.

Describe the solution you'd like
Ideally, we’d have a single target setting that specifies both modes: target: auto,75% would be a setting that: a) if the base coverage was below 75%, requires that each PR increases project coverage; and b) if the base coverage is over 75%, requires that each PR maintain coverage at or above 75%.

Such a setting would be relatively stable over the lifetime of a project; individual orgs could raise or lower the numerical part of the threshold to meet org-level quality goals; the level of testing will tend to move in the right direction when movement is appropriate; and dev velocity is not adversely impacted by coverage checks getting in the way.

Additional context
As coverage measurement is rolled out across the company, we expect projects to be in various stages of test maturity. Some projects are almost entirely untested; others already have >90% test coverage.

For relatively untested projects, setting a numerical coverage target isn’t useful: too high, and we’re blocking useful PRs even if they add unit tests; too low, and they don’t help to raise the bar for quality. We would like to encourage more testing, and would prefer to use auto targets.

For more mature projects, with already-high test coverage, an auto target gets in the way: a useful PR that drops coverage by 1% would be rejected even if we’re already at 90%. The higher the actual coverage, the more work it is to write a new PR. For such projects, we’d prefer a more static target, allowing users to relax their coverage so long as it stays “high enough”. (That last requirement of “high enough” is why the threshold specification isn’t helpful.)

[rohan-at-sentry]

Converting to discussion and looking for feedback on whether other uses feel this way and would find this useful