Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Line coverage metrics do not add up for executed_lines and summary.covered_lines in generated coverage.json #1900

Open
JohnPeng47 opened this issue Dec 2, 2024 · 8 comments
Labels
bug Something isn't working question Further information is requested

Comments

@JohnPeng47
Copy link

JohnPeng47 commented Dec 2, 2024

Describe the bug
Here below is an item in "files" generated from coverage.py:

"src\textual\widgets\_list_item.py": {
"executed_lines": [
1,
3,
5,
6,
7,
8,
11,
12,
19,
20,
22,
23,
25,
29,
32,
35,
36,
37
],
"summary": {
"covered_lines": 15,
"num_statements": 21,
"percent_covered": 71.42857142857143,
"percent_covered_display": "71",
"missing_lines": 6,
"excluded_lines": 0
},
"missing_lines": [
26,
27,
30,
33,
38,
39
],
"excluded_lines": [],

I would expect len(executed_lines) == summary.covered_lines but the first is 18 and the later is 15. Where does the extra 3 come from?

To Reproduce
I tried this with the following:
pytest . --tb short --color no --cov=src\textual --cov-report json --continue-on-collection-errors --disable-warnings
On the https://github.com/Textualize/textual repo.

Expected behavior
My expectation is:
covered_lines + executed_lines + excluded_lines == summary.num_statements, but this seems to not be the case?

@JohnPeng47 JohnPeng47 added the bug Something isn't working label Dec 2, 2024
@nedbat
Copy link
Owner

nedbat commented Dec 3, 2024

I'd like to reproduce this, but I don't use poetry so I'm not sure how to get everything in order. Can you give the exact commands you used, starting with git clone?

@nedbat nedbat added the question Further information is requested label Dec 3, 2024
@nedbat
Copy link
Owner

nedbat commented Dec 3, 2024

I think the three lines are the docstrings, but I'm not sure why they are treated differently.

@JohnPeng47
Copy link
Author

JohnPeng47 commented Dec 3, 2024

Yeah so here's the repo: https://github.com/JohnPeng47/textual-neutered

  1. Clone
  2. Install poetry in your current python env
  3. cd into repo and run poetry install to get deps
  4. run poetry shell to activate project venv with deps installed from 3
  5. run pytest . --tb short --color no --cov=src/textual --cov-report json --continue-on-collection-errors --disable-warnings
  6. [OPTIONAL] I have a script test_coverage.py that basically checks that this conditions is true for all coverage files:
    missing_check = len(missing_lines) + len(executed_lines) + len(excluded_lines) == total_stmts

@JohnPeng47
Copy link
Author

Also a shorter example here:

"__init__.py": {
  "executed_lines": [
    0
  ],
  "summary": {
    "covered_lines": 0,
    "num_statements": 0,
    "percent_covered": 100,
    "percent_covered_display": "100",
    "missing_lines": 0,
    "excluded_lines": 0
  },
  "missing_lines": [],
  "excluded_lines": [],

In this example, both covered_lines and num_statements should be 1 right, not 0?

@JohnPeng47
Copy link
Author

Also another thing that I just noticed now, is that the coverage collected on the same repo is different on Windows vs. Linux. Is this just inherent to the OS differences in settrace impl?

@nedbat
Copy link
Owner

nedbat commented Dec 4, 2024

There is no OS difference in the settrace implementation. Likely the difference is do to OS differences in textual repo. If you don't think that's the case, let's look at some more specific details.

@JohnPeng47
Copy link
Author

JohnPeng47 commented Dec 4, 2024

Ahh no I think that makes sense. But could you confirm:
len(missing_lines) + len(executed_lines) + len(excluded_lines) == total_stmts

Is invariant for every file under coverage?

@JohnPeng47
Copy link
Author

Hey sorry to keep pestering, but do you think you can just answer that one question above? My coverage diffing code is broken for some reason, and I just want to be sure that the above condition should hold

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants