Fix build deps compilation for setuptools < 70.1.0

[chrysle]

See #1681 (comment) for context.

Contributor checklist

• Included tests for the changes.
• PR title is short, clear, and ready to be included in the user-facing changelog.

Maintainer checklist

• Verified one of these labels is present: backwards incompatible, feature, enhancement, deprecation, bug, dependency, docs or skip-changelog as they determine changelog listing.
• Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

[webknjaz]

This is not a correct fix. It just shoves the problem under the carpet. We need to address the underlying problem, which is revealed not due to pip but because setuptools changed. Yet, the bug is in pip-tools: #2104 (comment). The correct change in the test is to include something like -P 'setuptools < 70.1.0'. But that should be accompanied with a fix for the bug.

[webknjaz]

Alternatively, it's even better to emulate such a situation with artificial package stubs so that it doesn't need to hit the network.

[chrysle]

@webknjaz I reworked the PR; please have a look. Of course there are still the failing Windows tests, can we just drop that horrible Microsoft product ;-)?

[WhyNotHugo]

Slightly changing the order of assertions should help see more details into the errors:

diff --git a/tests/test_cli_compile.py b/tests/test_cli_compile.py
index c5031fc..8db0038 100644
--- a/tests/test_cli_compile.py
+++ b/tests/test_cli_compile.py
@@ -787,13 +787,13 @@ def test_direct_reference_with_extras(runner):
             "pip-tools[testing,coverage] @ git+https://github.com/jazzband/[email protected]"
         )
     out = runner.invoke(cli, ["-n", "--rebuild", "--no-build-isolation"])
-    assert out.exit_code == 0
     assert (
         "pip-tools[coverage,testing] @ git+https://github.com/jazzband/[email protected]"
         in out.stderr
     )
     assert "pytest==" in out.stderr
     assert "pytest-cov==" in out.stderr
+    assert out.exit_code == 0
 
 
 def test_input_file_without_extension(pip_conf, runner):

Generally, I prefer to compare out.stderr before checking the status code in tests; if stderr isn't the expected output, pytest will print it when failing the assertion.

[webknjaz]

Generally, I prefer to compare out.stderr before checking the status code in tests; if stderr isn't the expected output, pytest will print it when failing the assertion.

That won't be necessary if you'll follow the principle of a single assertion per test. Then pytest will print out everything. But personally, I have --showlocals in all configs so that the context is always printed..

[webknjaz]

@chrysle the windows failure is because of the deprecation warning in pip, right?

[chrysle]

Could you elaborate on that? It seems that the resolver only found some pre-versions of a package, only I don't know much because the output is shortened.

[webknjaz]

Yeah, it's weird. Somebody needs to use https://github.com/marketplace/actions/debugging-with-tmate to see what's happening on Windows. The vars in CI seem to suggest that stderr is incomplete or something. Can it be some output buffering specific to windows? Do we need PYTHONUNBUFFERED=1 in tests?

[webknjaz]

I've converted this to draft to prevent accidental merging ahead of time. I'll undraft it once I'm done experimenting.

[webknjaz]

So apparently this is not being cleaned up: neither the env file, nor the file descriptor behind tempfile.NamedTemporaryFile() is closed.

This was mentioned as necessary in #1681 (comment).

The proper way to handle it would be to use a context manager.

[webknjaz]

FTR, I'm taking over this. I moved it into another CM locally. Will push shortly.

[webknjaz]

I'm going to squash a few commits, so here's the exact change for history:

diff --git a/piptools/build.py b/piptools/build.py
index f6f8ff6f..9b4b3a4d 100644
--- a/piptools/build.py
+++ b/piptools/build.py
@@ -184,6 +184,34 @@ def build_project_metadata(
         )
 
 
+@contextlib.contextmanager
+def _temporary_constraints_file_set_for_pip(
+    upgrade_packages: tuple[str, ...]
+) -> Iterator[None]:
+    sentinel = object()
+    original_pip_constraint = os.getenv("PIP_CONSTRAINT", sentinel)
+    pip_constraint_was_unset = original_pip_constraint is sentinel
+
+    with tempfile.NamedTemporaryFile(mode="w+t") as tmpfile:
+        # Write packages to upgrade to a temporary file to set as
+        # constraints for the installation to the builder environment,
+        # in case build requirements are among them
+        tmpfile.write("\n".join(package for package in upgrade_packages))
+        tmpfile.flush()
+        os.environ["PIP_CONSTRAINT"] = tmpfile.name
+        try:
+            yield
+        finally:
+            if pip_constraint_was_unset:
+                del os.environ["PIP_CONSTRAINT"]
+                return
+
+            # Assert here is necessary because MyPy can't infer type
+            # narrowing in the complex case.
+            assert isinstance(original_pip_constraint, str)
+            os.environ["PIP_CONSTRAINT"] = original_pip_constraint
+
+
 @contextlib.contextmanager
 def _create_project_builder(
     src_dir: pathlib.Path,
@@ -201,16 +229,13 @@ def _create_project_builder(
         yield build.ProjectBuilder(src_dir, runner=runner)
         return
 
-    if upgrade_packages is not None:
-        # Write packages to upgrade to a temporary file to set as
-        # constraints for the installation to the builder environment,
-        # in case build requirements are among them
-        tmpfile = tempfile.NamedTemporaryFile(mode="w+t", delete=False)
-        tmpfile.write("\n".join(package for package in upgrade_packages))
-        tmpfile.flush()
-        os.environ["PIP_CONSTRAINT"] = tmpfile.name
+    maybe_pip_constrained_context = (
+        contextlib.nullcontext()
+        if upgrade_packages is None
+        else _temporary_constraints_file_set_for_pip(upgrade_packages)
+    )
 
-    with build.env.DefaultIsolatedEnv() as env:
+    with maybe_pip_constrained_context, build.env.DefaultIsolatedEnv() as env:
         builder = build.ProjectBuilder.from_isolated_env(env, src_dir, runner)
         env.install(builder.build_system_requires)
         env.install(builder.get_requires_for_build("wheel"))

[webknjaz]

So, while debugging via tmate, I discovered that on Windows, this raises a SubprocessError from somewhere within pypa/build. I'm not yet sure what's happening. The command was pip install --pep517something.. I didn't get to save it since the SSH session expires in about 40 minutes and I just lost it. I think, it's https://github.com/pypa/build/blob/88ae833/src/build/env.py#L257-L265.

This then causes the visible error we saw in the logs — [RuntimeError("generator didn't yield")](https://github.com/python/cpython/blob/6bc3e83/Lib/contextlib.py#L143). I'm not entirely sure why, though — might be a CPython bug.

Now that I'm thinking about it, I'm curious if the order of the CMs in the with statement matters. Perhaps, build.env.DefaultIsolatedEnv() gets interrupted and processes the __exit__() first, suppressing the error. Then, the control flow of _create_project_builder() is interrupted, never reaching the yield, tripping the validation in the outer @contextlib.contextmanager() decorator.
Though, I don't understand why SubprocessError doesn't escape the with block… can it be https://github.com/python/cpython/blob/6bc3e83/Lib/contextlib.py#L163-L167?

[webknjaz]

Calling _create_project_builder() or build_project_metadata() doesn't seem to be enough to reproduce this. I suspect, Click might be the cause.

[webknjaz]

UPD: it looks like pyproject.toml stubs in the tests that don't have a [build-system] section cause build to crash on Windows trying to access the setuptools.build_meta:__legacy__ fallback.

[webknjaz]

Oh, it looks like specifying it manually makes another module non-importable: out.stderr="Backend 'setuptools.build_meta' is not available.\nFailed to parse C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-unknown\\pytest-0\\popen-gw2\\test_compile_build_targets_set0\\pyproject.toml\n". So that ain't it.
I bet there's some escaping issue with Windows paths and/or PYTHONPATH etc.

[webknjaz]

(documenting the troubleshooting env setup procedure)

Here's what I do upon connection to the windows job over tmate (== tmux over SSH):

.tox/piplatest-coverage/Scripts/pip install -e .
.tox/piplatest-coverage/Scripts/pytest --no-cov -n0 -svvvvv -l tests/test_cli_compile.py::test_compile_build_targets_setuptools_no_wheel_dep
.tox/piplatest-coverage/Scripts/pip-compile.exe --build-deps-for wheel C:/msys64/tmp/pytest-of-runneradmin/pytest-0/test_compile_build_targets_set0/pyproject.toml --output-file - -vvvvvv

This lets me populate the temporary files for inspection (the pytest command). It also allows me to edit files in-place (Ctrl+b c to create a new tmux tab; Ctrl+b n / Ctrl+b p / Ctrl+b 1 / Ctrl+b 2 to switch tabs) and rerun pytest or pip-compile directly.

Editing files is done via nano since vim does not play well with that windows/cygwin env, I haven't checked why.

Scrolling the terminal output: Ctrl+b PgUp.

[webknjaz]

.tox/piplatest-coverage/Scripts/pip-compile.exe --build-deps-for wheel C:/msys64/tmp/pytest-of-runneradmin/pytest-0/test_compile_build_targets_set0/pyproject.toml --output-file - -vvvvvv

Better repro without running pytest upfront:

$ .tox/piplatest-coverage/Scripts/pip-compile.exe --build-deps-for wheel tests/test_data/packages/small_fake_with_pyproject/pyproject.toml --output-file - -vvvvvv

[webknjaz]

I verified that setting the PIP_CONSTRAINT env var is exactly what breaks on Windows. Commenting it out makes the command pass.

[webknjaz]

# test-venv/Scripts/pip-compile --build-deps-for wheel tests/test_data/packages/small_fake_with_pyproject/pyproject.toml --output-file - -vvvvvv
Using pip-tools configuration defaults found in 'pyproject.toml'.
Creating isolated environment: venv+pip...
tmpfile.name='C:\\msys64\\tmp\\tmpti95poux'
Installing packages in isolated environment:
- setuptools < 70
Getting metadata for wheel...
Backend 'setuptools.build_meta' is not available.
Failed to parse D:\a\pip-tools\pip-tools\tests\test_data\packages\small_fake_with_pyproject\pyproject.toml

I wonder if reversing the slashes is needed or something.

[webknjaz]

Ah-ha!

I think I understand what's happening now! The temporary file is open and flushed, but the file descriptor remains open, which on Windows prevents other processes from accessing it. So when we call build, it calls pip install that sees PIP_CONSTRAINT, attempts to open it, and likely fails to install the build deps because of permission denied.

It seems like calling .close() upon .flush() (or should it be instead, even?) makes it accessible. We just have to use delete_on_close=False so that .close() does not also remove the file from disk right away. With delete=True (which is the default), the CM will clean it up on exit.

Some of this is mentioned @ https://docs.python.org/3/library/tempfile.html#tempfile.NamedTemporaryFile

[webknjaz]

Urgh.. looks like delete_on_close got introduced in Python 3.12. At least I now have an idea of a workaround. Will try to complete this PR today.

[webknjaz]

TODO: check if moving this back into the outer CM works.

[webknjaz]

This will be a part of the CI fix, I excluded the pip pinning commit as it will be coming from #2142.

Test run with pip pins: https://github.com/jazzband/pip-tools/actions/runs/12292796650.

Please do not add any changes to this PR.