Coverage lines on pull requests are incorrect

[MetRonnie]

Describe the bug

When a GitHub Actions workflow runs on the pull_request event, actions/checkout by default does not check out the pull request branch itself, but merges it into the base branch and checks out this merge commit.

This means that the reported coverage is for this merge commit, but the Codecov website does not account for this. When you view the coverage for a PR on the Codecov website, it shows the files as they are in the PR branch, meaning the coverage lines are incorrect (when there have been other changes to the same files committed to the base branch).

Environment (please complete the following information):

• Browser N/A
• System N/A

To Reproduce
Steps to reproduce the behavior:

1. Look at https://app.codecov.io/gh/cylc/cylc-flow/pull/6445/blob/cylc/flow/task_pool.py#L879 (screenshot below)
2. Note that a different PR was merged into the base branch in between the base & head commits for this PR: cylc/cylc-flow@fb75928#diff-becd22b70a92532763c24d9e0bedde95e36801c2eac01a469939f006c2faa148R880

Expected behavior
I guess the only solution is for the Codecov website to display the same internal merge commit that actions/checkout checked out, rather than the head commit of the PR

Screenshots

Here the coverage lines start to go wrong at L880 - executable lines are shown as unexecutable and vice versa!

Here is the commit graph. The highlighted merge commit added some lines at L880 to the same file.

Here is that merge commit.

Additional context
Here is the GitHub Actions workflow for this repo: https://github.com/cylc/cylc-flow/blob/2f495817a19f09f4ed12aa6a00162a2923b291eb/.github/workflows/test_fast.yml

[michelletran-codecov]

Sorry for the delay in responding. @MetRonnie It looks like the linked issue is no longer inconsistent. I suspect that the problem was related to the coverage report still being processed when accessing the webpage. We are working on a longer term solution in this issue: codecov/engineering-team#2702

[MetRonnie]

You're right that the coverage I linked is now correct, as it includes the few lines added on the base branch by my colleague. However I think this is because I rebased and force-pushed the branch after opening this issue

[michelletran-codecov]

It sounds like rebasing solved the issue for this PR. The larger systemic issue is still likely related to a lag in coverage reporting (as linked in a ticket above) or coverage processing failures. Closing this as it is no longer an issue for the linked issue. If you run into this issue again, feel free to create a new ticket.