Mark known test failures

[macro1]

Marks known test failures tied to #2131 as we do not have a strategy for remediating.

Contributor checklist

• Included tests for the changes.
• PR title is short, clear, and ready to be included in the user-facing changelog.

Maintainer checklist

• Verified one of these labels is present: backwards incompatible, feature, enhancement, deprecation, bug, dependency, docs or skip-changelog as they determine changelog listing.
• Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

[webknjaz]

IIRC, the fix is #2106. I'd rather accept it instead.

[webknjaz]

While I don't think that the tests should be suppressed at all, if we go this way, the places mentioned below must be fixed first.

[macro1]

@webknjaz we either suppress or halt development on the project. there's no proposed solution for #2110 #2131

edit: 🤦 i tied everything to the wrong issue. i meant #2131

[webknjaz]

we either suppress or halt development on the project. there's no proposed solution for #2131

I guess you could work on the solution to that issue with a few temporary commits that unblock the CI for you and could be dropped later.

Meanwhile, the correct solution to the failing tests is #2106. Fixing it is what has to happen first.

I'm blocking this because the change at https://github.com/jazzband/pip-tools/pull/2137/files#r1867006850 is harmful in that it contributes to shoving the problem under the rug, which is a big deal.

Since nobody seems to be attempting to look into completing #2106, I'll try to check it myself later today.

[macro1]

alright. so we'll rely on #2106 for the wheel expected in output issue, #2134 for the pre-commit issue (the mypy step no longer supports 3.8 runtime). in this PR, i'll narrow it down to the known issue introduced in pip 24.3

i'm curious about #2106 and the problems we would potentially be shoving beneath the rug, so i may comment there later when i have more time

[webknjaz]

I've converted this to draft to prevent accidental merging ahead of time.

[macro1]

@webknjaz currently this PR is blocked by your request for changes, and now it is also a draft, so i don't think it's going anywhere

what needs to be merged ahead of this PR? the changes have been reduced to marking specific tests as expected failures when using a pip version of 24.3 or higher, which will remain true till there's a solution for #2131

[webknjaz]

@macro1 in #2106, I'm experimenting with making the testing more predictable by pinning pip more aggressively in the test infra (#2106 (comment)). I recall that PR failing weirdly on Windows, and now that's not exactly the case.

Now that I identified this, I'm going to see if I can change that setup together with #2134 and make pip updates in CI more controlled.

[webknjaz]

@macro1 this is not technically necessary anymore. I've still clicked “rebase” to get the CI statuses to me green. Though, perhaps this should be updated together with bumping the pip version that I pinned in another PR. And with the corresponding fixes instead of disabling the tests...

[webknjaz]

@macro1 let me know if you think this should be merged anyway.

[macro1]

@webknjaz i think i mentioned before there's no current solution for the failing tests this PR is set to mark as known failures. you're currently skipping the tests by limiting the versions of pip we run any tests for, whereas this PR was specifically targeting the known failures.

imo introducing the idea of supported versions of pip runs contrary to how people expect to use this project, but it's a difference of philosophy and probably won't matter in the long-run.

i disagree and do not think it's my choice on whether to merge these changes in at this point. we now have the concept of supported pip versions.. if that is sufficient, that is the approach you have led us to. in my opinion, you've made the choice for us on which approach to take

i do want you to know i'm really happy CI is able to produce green statuses again, and it's awesome there are active contributors here

[webknjaz]

@macro1 I asked you if you wanted to merge because I think your opinion is important too. I'm kinda open to it but it might make sense to couple it with bumping pip.

As for "supported", Hugo raised a concern in another PR. We talked about renaming it but postponed, prioritizing making the PR green. It turned out to be confusing but I did not mean to imply that everything else is unsupported. I just wanted to have something that the PR authors would be able to rely on and could be bumped in a controlled and transparent way.
In the metadata, this is not limited.
We should probably rename it to "latest tested".

[macro1]

including this change when adding the other pip versions to testing makes sense to me