Tracking issue for function attribute `#[coverage]` #84605

richkadel · 2021-04-27T01:35:11Z

This issue will track the approval and stabilization of the attribute coverage, needed to give developers a way to "hide" a function from the coverage instrumentation enabled by rustc -Z instrument-coverage.

The Eq trait in the std library implements a marker function that is not meant to be executed, but results in an uncovered region in all rust programs that derive Eq. This attribute will allow the compiler to skip that function, and remove the uncovered regions.

Unresolved questions

is it ok that the attribute applied to the crate root, a function, or a module applies to all items within the marked item? So far we do not have such attributes other than the lint level attributes.

The text was updated successfully, but these errors were encountered:

richkadel · 2021-04-27T01:35:26Z

cc: @tmandry @wesleywiser

nagisa · 2021-04-27T12:27:34Z

Nit: should probably follow the pattern of #[coverage(...)] much like #[inline] and many other attributes.

Swatinem · 2021-04-27T12:37:40Z

Just a wish that I have:
Instead of making this a function level attr, it should work equally on mod or even just blocks.

Another thing I wish for would be a crate-level attr that would disable coverage for every #[test] fn.

Adds feature-gated `#[no_coverage]` function attribute, to fix derived Eq `0` coverage issue rust-lang#83601 Derived Eq no longer shows uncovered The Eq trait has a special hidden function. MIR `InstrumentCoverage` would add this function to the coverage map, but it is never called, so the `Eq` trait would always appear uncovered. Fixes: rust-lang#83601 The fix required creating a new function attribute `no_coverage` to mark functions that should be ignored by `InstrumentCoverage` and the coverage `mapgen` (during codegen). Adding a `no_coverage` feature gate with tracking issue rust-lang#84605. r? `@tmandry` cc: `@wesleywiser`

richkadel · 2021-04-28T20:09:57Z

Nit: should probably follow the pattern of #[coverage(...)] much like #[inline] and many other attributes.

I considered this, and discussed it in the initial PR. I'll copy those comments to this tracking issue, to make it easer to read and reply:

@tmandry said:

I also like the idea of combining the first two comments on #84605 by making it #[coverage(always/never)] and making some things (like tests?) disabled by default, but #[coverage(always)] could be used to override this. The only drawbacks I see are that #[coverage] on its own seems meaningless (unlike #[inline]) and these are longer to type.

The particulars of which things to make covered by default are a little hazy to me at this point, though. Really the problem I want to fix is that assert messages in tests appear uncovered, but that's kind of the point :)

@richkadel replied:

I also like the idea of combining the first two comments on #84605 by making it #[coverage(always/never)] and making some things (like tests?) disabled by default

Yes, I saw the comments, and I do want to consider them. Since this is feature-gated (and fixes a bug), I'd like to land this with no_coverage initially, and follow up with a different PR if we decide to change how it works.

I did consider something like #[coverage(never)], but (as you point out) I currently feel that #[coverage] or #[coverage(always)] doesn't really make sense to me.

In fact, I think coverage is especially relevant to tests, so I wouldn't want to disable it for tests by default, or require the user add the attribute in order to get test coverage.

The only other coverage-related attribute we may want to consider, in the future, is an attribute to enable coverage of unwind paths (#78544). But the MIR InstrumentCoverage pass isn't close to ready to support that yet.

It's definitely not uncommon for attributes to start with no_, and I just felt that no_coverage is much easier to understand and easier to remember compared to coverage(never) or coverage(none) or coverage(disable). (Also, never might be consistent with the terminology for inline, but the semantics are very different. inline has the notion of "best effort" or "when appropriate" inlining; but those semantics don't apply as well to coverage I think.)

@tmandry replied:

Agreed that coverage is relevant to tests, but I don't know if it's very relevant to the test functions themselves. But like I said, there might be a different way of getting at the problem I'm talking about.

In fact, your reply made me think of an approach I like better, which is removing counters that are along a path that always panics. We don't instrument the full panic paths anyway, and this is a straightforward analysis to do.

None of this is very relevant to this PR though! I just wanted to put my ideas down somewhere..

richkadel · 2021-04-28T20:27:49Z

Just a wish that I have:
Instead of making this a function level attr, it should work equally on mod or even just blocks.

Another thing I wish for would be a crate-level attr that would disable coverage for every #[test] fn.

@Swatinem - Can you expand on this? What are the use cases for disabling coverage at the crate level, or in crate tests?

For example, the use case addressed in PR #83601 is clear: The implementation of Eq required adding a special marker function (which gets added in every struct that derives from Eq) that was not intended to be called. Every struct that derived from Eq could never achieve 100% coverage. This unexecutable function clearly justifies the requirement for #[no-coverage].

I personally see the Eq trait issue as an exceptional case.

Generally speaking...

I think we want to keep the bar high for now (by restricting the scope of the attribute to functions, until any broader requirement is vetted by the Rust community).

So if there are other strong cases for excluding coverage instrumentation, please post them here so we can validate the use cases before expanding the scope of the coverage attribute.

richkadel · 2021-04-28T20:31:10Z

One more comment for clarity: PR #83601 has been merged and adds the feature-gated #[no_coverage] attribute. The attribute can be enabled on an individual function (by also adding #[feature(no_coverage]) or at the crate level.

richkadel · 2021-04-28T20:34:31Z

I should have added: This tracking issue will remain open until the attribute is stablized. This means there is still an opportunity to change the attribute, based on the tracking issue feedback.

Thanks!

Swatinem · 2021-04-29T10:28:59Z

@Swatinem - Can you expand on this? What are the use cases for disabling coverage at the crate level, or in crate tests?

This might be a bit personal preference, but is also done that way in other language ecosystems.
You generally want to have coverage for the code under test, not the test code itself. Having coverage metrics for test code provides zero value to me as developer.

nagisa · 2021-04-29T11:18:35Z

Note: there are previous initiatives to introduce inheritable/scoped/propagated attributes, e.g. for the optimize attribute, but implementing the propagation behaviour there seemed to be pretty involved to me at the time and IMO any attempts to introduce such attribute propagation mechanisms should be a standalone RFC introducing a mechanism in a more general way.

72: Set cfg(coverage) to easily use #[no_coverage] r=taiki-e a=taiki-e To exclude the specific function from coverage, use the `#[no_coverage]` attribute (rust-lang/rust#84605). Since `#[no_coverage]` is unstable, it is recommended to use it together with `cfg(coverage)` set by cargo-llvm-cov. ```rust #![cfg_attr(coverage, feature(no_coverage))] #[cfg_attr(coverage, no_coverage)] fn exclude_from_coverage() { // ... } ``` Closes #71 Co-authored-by: Taiki Endo <[email protected]>

jhpratt · 2021-09-08T09:27:47Z

Would it be possible to add this attribute to the code generated by unreachable!()? Behind a flag, of course. Both #![feature(no_coverage)] and LLVM-based code coverage are unstable, so I don't think this would cause any issues unless I'm missing something.

richkadel · 2021-09-08T16:40:30Z

Unfortunately, #[no_coverage] (in its current form) applies to a function definition only. Applying that attributed before a function definition will cause the entire function to be ignored by coverage. But I assume you want the "call" to unreachable!() to be ignored by coverage. Ignoring a statement or block is not currently supported.

I can understand the motivation for this request, and you may want to file a new issue, if it doesn't already exist. There is a potential downside: coverage reports would not be able to show that an unreachable block was unintentionally executed. Maybe that's a minor tradeoff, but the current coverage reports confirm that unreachable blocks are not reached, by showing the unreachable block has a coverage count of zero.

I think adding support for #[no_coverage] blocks may be the best solution, but macro expansion may make this tricky to get right. There may not be a quick solution.

richkadel · 2021-09-08T16:49:13Z

And I may be wrong about block #[no_coverage] being the "best solution". Thinking about this just a bit more, adding the #[no_coverage] attribute at the function level was not that difficult, because AST-level functions map directly to MIR bodies (and coverage is instrumented within each MIR body). That's easy to turn on or off at the function level. But AST-level blocks don't translate directly to internal MIR structure. That will also make implementing more granular support difficult, in addition to macro expansion complications.

tmandry · 2021-09-08T22:02:09Z

To me, #[no_coverage] blocks seem like the best solution from a language perspective. Agreed that there might be some work threading things through on the implementation side.

For macros, we can just think about this as applying to a particular expansion (and the attribute applies to the block in HIR, i.e. after macro expansion). Maybe there are other issues I'm not thinking about.

jhpratt · 2021-09-08T23:03:57Z

I don't think accidentally executing an "unreachable" expression is an issue for coverage, as that's not the job of coverage; tests are where that should be caught.

There is a tracking issue for this already, I wasn't sure if there was an easy-ish way to land it. Apparently that's not the case. I would imagine in the future statement and expression-level masking for coverage purposes will exist, which would necessarily allow for this. Ultimately this would just cause the macro to include the attribute in its expansion.

clarfonthey · 2021-12-02T04:41:04Z

Is there a good reason why derived traits are included in coverage reports, and not also marked as #[no_coverage]? Considering how the assumption for derived traits is that they're already "tested" by the standard library, I figure that it makes the most sense to simply not include coverage for them by default.

But, I could see an argument for including them by default but allowing a #[no_coverage] attribute on struct definitions to disable them, if you feel like going that route.

clarfonthey · 2021-12-02T05:19:34Z

In terms of specifically how this attribute should be named, I would be in favour of going with a #[coverage(...)] style attribute as @nagisa recommended. This also opens up adding other coverage-based attributes in the future, like:

#[coverage(hide)] -- the existing #[no_coverage] attribute
#[coverage(ignore)] -- would mark code as "coverage doesn't matter," e.g. derives, telling tools to still show coverage for lines but not including it in total coverage metrics
#[coverage(disallow)] -- would mark code as "should not be covered", e.g. the unreachable!() macro, allowing tools to show warnings when they are covered
#[coverage(require)] -- would mark code as "should always be covered", allowing tools to show warnings when they are not covered

This obviously wouldn't be required for an initial stable #[coverage] attribute, but I think in particular the distinction between #[coverage(hide)] and #[coverage(ignore)] might be useful to have. Something like the Eq method should be marked #[coverage(hide)] because it's more confusing to show coverage than not, but in general I think it would make sense to mark other stuff as #[coverage(ignore)] in general.

richkadel · 2021-12-02T15:20:04Z

I'm not convinced ignoring trait functions is a good idea, from a test coverage perspective.

When you derive a trait, your struct's API contract has changed. That struct now supports every function defined in that trait. To guarantee 100% test coverage of the behaviors of every function for your struct, you would have to test it.

Without adding a test, the "ignore" concept is really just a cheat, because you want to achieve 100% test coverage without actually testing it 100%. You can say you are "really confident" that it will work fine, but you can't prove it without some kind of test analysis or observed behavior, via a test.

clarfonthey · 2021-12-02T21:14:33Z

Yes, the API contract changes when you derive standard traits, but I personally don't believe that testing them adds any value for the developer. For example, assert_eq!(value, value.clone()); would provide coverage for derived PartialEq and Clone implementations, but as far as your library goes, you're not really testing anything important; the only way those would fail is if there's an error in the compiler, which I don't think is everyone's responsibility to test.

I should clarify I'm explicitly talking about the standard derives; individual proc macros can make the decision whether they want to add the attribute to their generated code or not.

PoignardAzur · 2024-09-18T14:03:41Z

This might come late in the discussion, but have you considered stabilizing coverage as a diagnostic namespace attribute? E.g.:

#[diagnostic::coverage(off)]
fn foobar() {}

The main advantage of doing that is that people could then use coverage annotations while keeping a MSRV of 1.78.0. Same thing for future changes to the feature, if any.

And I do think coverage as a diagnostic makes sense. Yes, it changes the emitted artifact, but it doesn't change the program semantics. And more importantly, it doesn't affect dependencies: if a dependency has coverage annotations you don't understand, you can just ignore them without negative impact to your output.

clarfonthey · 2024-09-18T15:05:38Z

Coverage isn't a diagnostic, though; it's a build flag that can change the way the crate is built under certain circumstances.

…e, r=wesleywiser Stabilize #[coverage] attribute Closes rust-lang#84605, which passed FCP. Stabilisation report here: rust-lang#84605 (comment) Also added to reference here: rust-lang/reference#1628

…e, r=<try> Stabilize #[coverage] attribute Closes rust-lang#84605, which passed FCP. Stabilisation report here: rust-lang#84605 (comment) Also added to reference here: rust-lang/reference#1628 --- try-job: aarch64-apple try-job: x86_64-gnu try-job: x86_64-msvc

ehuss · 2024-12-18T22:53:19Z

@clarfonthey Are there any tests for this behavior:

Note: traits with this attribute apply only to their default method bodies, and does not affect implementations at all.

clarfonthey · 2024-12-19T04:11:48Z

I assume not, since it would be much more difficult to implement them applying to all downstream implementations since those are, effectively, completely different functions and completely different codegen as far as instrumentation is concerned.

If you think a test should exist, I would file a new issue. Unless we plan to revert the stabilisation before release, I think that we should just open new issues to track new changes to this.

ehuss · 2024-12-19T14:52:35Z

I opened a PR to add the tests at #134517

…esleywiser Revert stabilization of the `#[coverage(..)]` attribute Due to a process mixup, the PR to stabilize the `#[coverage(..)]` attribute (rust-lang#130766) was merged while there are still outstanding concerns. The default action in that situation is to revert, and the feature is not sufficiently urgent or uncontroversial to justify special treatment, so this PR reverts that stabilization. --- - A key point that came up in offline discussions is that unlike most user-facing features, this one never had a proper RFC, so parts of the normal stabilization process that implicitly rely on an RFC break down in this case. - As the implementor and de-facto owner of the feature in its current form, I would like to think that I made good choices in designing and implementing it, but I don't feel comfortable proceeding to stabilization without further scrutiny. - There hasn't been a clear opportunity for T-compiler to weigh in or express concerns prior to stabilization. - The stabilization PR cites a T-lang FCP that occurred in the tracking issue, but due to the messy design and implementation history (and lack of a clear RFC), it's unclear what that FCP approval actually represents in this case. - At the very least, we should not proceed without a clear statement from T-lang or the relevant members about the team's stance on this feature, especially in light of the other concerns listed here. - The existing user-facing documentation doesn't clearly reflect which parts of the feature are stable commitments, and which parts are subject to change. And there doesn't appear to be a clear consensus anywhere about where that line is actually drawn, or whether the chosen boundary is acceptable to the relevant teams and individuals. - For example, the [stabilization report comment](rust-lang#84605 (comment)) mentions that some aspects are subject to change, but that text isn't consistent with my earlier comments, and there doesn't appear to have been any explicit discussion or approval process. - [The current reference text](https://github.com/rust-lang/reference/blob/4dfaa4f/src/attributes/coverage-instrumentation.md) doesn't mention this distinction at all, and instead simply describes the current implementation behaviour. - When the implementation was changed to its current form, the associated user-facing error messages were not updated, so they still refer to the attribute only being allowed on functions and closures. - On its own, this might have been reasonable to fix-forward in the absence of other concerns, but the fact that it never came up earlier highlights the breakdown in process that has occurred here. --- Apologies to everyone who was excited for this stabilization to land, but unfortunately it simply isn't ready yet.

…eywiser Stabilize #[coverage] attribute Closes #84605, which passed FCP. Stabilisation report here: rust-lang/rust#84605 (comment) Also added to reference here: rust-lang/reference#1628 --- try-job: aarch64-apple try-job: x86_64-gnu try-job: x86_64-msvc

Rollup merge of rust-lang#134672 - Zalathar:revert-coverage-attr, r=wesleywiser Revert stabilization of the `#[coverage(..)]` attribute Due to a process mixup, the PR to stabilize the `#[coverage(..)]` attribute (rust-lang#130766) was merged while there are still outstanding concerns. The default action in that situation is to revert, and the feature is not sufficiently urgent or uncontroversial to justify special treatment, so this PR reverts that stabilization. --- - A key point that came up in offline discussions is that unlike most user-facing features, this one never had a proper RFC, so parts of the normal stabilization process that implicitly rely on an RFC break down in this case. - As the implementor and de-facto owner of the feature in its current form, I would like to think that I made good choices in designing and implementing it, but I don't feel comfortable proceeding to stabilization without further scrutiny. - There hasn't been a clear opportunity for T-compiler to weigh in or express concerns prior to stabilization. - The stabilization PR cites a T-lang FCP that occurred in the tracking issue, but due to the messy design and implementation history (and lack of a clear RFC), it's unclear what that FCP approval actually represents in this case. - At the very least, we should not proceed without a clear statement from T-lang or the relevant members about the team's stance on this feature, especially in light of the other concerns listed here. - The existing user-facing documentation doesn't clearly reflect which parts of the feature are stable commitments, and which parts are subject to change. And there doesn't appear to be a clear consensus anywhere about where that line is actually drawn, or whether the chosen boundary is acceptable to the relevant teams and individuals. - For example, the [stabilization report comment](rust-lang#84605 (comment)) mentions that some aspects are subject to change, but that text isn't consistent with my earlier comments, and there doesn't appear to have been any explicit discussion or approval process. - [The current reference text](https://github.com/rust-lang/reference/blob/4dfaa4f/src/attributes/coverage-instrumentation.md) doesn't mention this distinction at all, and instead simply describes the current implementation behaviour. - When the implementation was changed to its current form, the associated user-facing error messages were not updated, so they still refer to the attribute only being allowed on functions and closures. - On its own, this might have been reasonable to fix-forward in the absence of other concerns, but the fact that it never came up earlier highlights the breakdown in process that has occurred here. --- Apologies to everyone who was excited for this stabilization to land, but unfortunately it simply isn't ready yet.

Add tests for coverage attribute on trait functions This adds tests for the coverage attribute on trait functions. cc rust-lang#84605 (comment)

Zalathar · 2024-12-23T14:07:00Z

This was auto-closed by #130766, but that has since been reverted (#134672) due to outstanding concerns.

This remains the canonical tracking issue for the coverage attribute, but since it's long and unwieldy I'm thinking of opening a separate issue to focus directly on the remaining steps, and linking it from the top of this issue page.

EDIT: New stabilization tracking sub-issue is #134749. This issue (#84605) remains the “primary” tracking issue for the feature overall.

traviscross · 2024-12-23T14:18:05Z

@rfcbot cancel

As noted in the revert, our FCP on this is now kind of ancient and should be reclarified. Whenever this is later proposed again for stabilization, please nominate the stabilization PR for lang for proper handling.

rfcbot · 2024-12-23T14:18:10Z

@traviscross proposal cancelled.

Rollup merge of rust-lang#134517 - ehuss:coverage-trait-test, r=Zalathar Add tests for coverage attribute on trait functions This adds tests for the coverage attribute on trait functions. cc rust-lang#84605 (comment)

richkadel mentioned this issue Apr 27, 2021

Adds feature-gated #[no_coverage] function attribute, to fix derived Eq 0 coverage issue #83601 #84562

Merged

Mark-Simulacrum mentioned this issue May 2, 2021

no_coverage feature-gated on function, not crate? #84836

Closed

Bryysen mentioned this issue Jun 11, 2021

Warning when specifying the output name with cargo rustc rust-lang/cargo#9558

Open

jhpratt mentioned this issue Aug 26, 2021

Question: How do you exclude a line from the coverage report? taiki-e/cargo-llvm-cov#71

Closed

taiki-e mentioned this issue Aug 26, 2021

Set cfg(coverage) to easily use #[no_coverage] taiki-e/cargo-llvm-cov#72

Merged

richkadel mentioned this issue Oct 21, 2021

Stabilize -Z instrument-coverage as -C instrument-coverage #90132

Merged

This was referenced Sep 23, 2024

coverage attribute rust-lang/reference#1628

Merged

Stabilize #[coverage] attribute #130766

Merged

mnemonikr mentioned this issue Oct 11, 2024

Improving tests and documentation in memory module mnemonikr/symbolic-pcode#137

Merged

taiki-e mentioned this issue Dec 4, 2024

Exclude instantiations from statistics? taiki-e/cargo-llvm-cov#397

Closed

bors closed this as completed in 1d35638 Dec 17, 2024

ehuss mentioned this issue Dec 19, 2024

Add tests for coverage attribute on trait functions #134517

Merged

Zalathar mentioned this issue Dec 23, 2024

Revert stabilization of the #[coverage(..)] attribute #134672

Merged

Zalathar reopened this Dec 23, 2024

rfcbot removed the disposition-merge This issue / PR is in PFCP or FCP with a disposition to merge it. label Dec 23, 2024

traviscross removed finished-final-comment-period The final comment period is finished for this PR / Issue. S-tracking-ready-to-stabilize Status: This is ready to stabilize; it may need a stabilization report and a PR labels Dec 23, 2024

Zalathar mentioned this issue Dec 25, 2024

Tracking issue for stabilizing the #[coverage(..)] attribute #134749

Open

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tracking issue for function attribute `#[coverage]` #84605

Tracking issue for function attribute `#[coverage]` #84605

richkadel commented Apr 27, 2021 •

edited by oli-obk

Loading

richkadel commented Apr 27, 2021

nagisa commented Apr 27, 2021

Swatinem commented Apr 27, 2021

richkadel commented Apr 28, 2021

richkadel commented Apr 28, 2021

richkadel commented Apr 28, 2021

richkadel commented Apr 28, 2021

Swatinem commented Apr 29, 2021

nagisa commented Apr 29, 2021

jhpratt commented Sep 8, 2021

richkadel commented Sep 8, 2021

richkadel commented Sep 8, 2021

tmandry commented Sep 8, 2021

jhpratt commented Sep 8, 2021

clarfonthey commented Dec 2, 2021

clarfonthey commented Dec 2, 2021

richkadel commented Dec 2, 2021

clarfonthey commented Dec 2, 2021 •

edited

Loading

PoignardAzur commented Sep 18, 2024 •

edited

Loading

clarfonthey commented Sep 18, 2024

ehuss commented Dec 18, 2024

clarfonthey commented Dec 19, 2024

ehuss commented Dec 19, 2024

Zalathar commented Dec 23, 2024 •

edited

Loading

traviscross commented Dec 23, 2024 •

edited

Loading

rfcbot commented Dec 23, 2024

Tracking issue for function attribute #[coverage] #84605

Tracking issue for function attribute #[coverage] #84605

Comments

richkadel commented Apr 27, 2021 • edited by oli-obk Loading

Unresolved questions

richkadel commented Apr 27, 2021

nagisa commented Apr 27, 2021

Swatinem commented Apr 27, 2021

richkadel commented Apr 28, 2021

richkadel commented Apr 28, 2021

richkadel commented Apr 28, 2021

richkadel commented Apr 28, 2021

Swatinem commented Apr 29, 2021

nagisa commented Apr 29, 2021

jhpratt commented Sep 8, 2021

richkadel commented Sep 8, 2021

richkadel commented Sep 8, 2021

tmandry commented Sep 8, 2021

jhpratt commented Sep 8, 2021

clarfonthey commented Dec 2, 2021

clarfonthey commented Dec 2, 2021

richkadel commented Dec 2, 2021

clarfonthey commented Dec 2, 2021 • edited Loading

PoignardAzur commented Sep 18, 2024 • edited Loading

clarfonthey commented Sep 18, 2024

ehuss commented Dec 18, 2024

clarfonthey commented Dec 19, 2024

ehuss commented Dec 19, 2024

Zalathar commented Dec 23, 2024 • edited Loading

traviscross commented Dec 23, 2024 • edited Loading

rfcbot commented Dec 23, 2024

Tracking issue for function attribute `#[coverage]` #84605

Tracking issue for function attribute `#[coverage]` #84605

richkadel commented Apr 27, 2021 •

edited by oli-obk

Loading

clarfonthey commented Dec 2, 2021 •

edited

Loading

PoignardAzur commented Sep 18, 2024 •

edited

Loading

Zalathar commented Dec 23, 2024 •

edited

Loading

traviscross commented Dec 23, 2024 •

edited

Loading