build: Add setuptools to Dockerfile #11358
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
product-auto-label
bot
added
the
samples
kweinmeister
added
the
kokoro:force-run
kokoro-team
removed
the
kokoro:force-run
parthea
approved these changes
Mar 23, 2024
| RUN pip install --no-cache-dir virtualenv | ||
| # Install "setuptools" for Python 3.12+ (see https://docs.python.org/3/whatsnew/3.12.html#distutils) | ||
| # Install "virtualenv", since the vast majority of users of this image will want it. | ||
| RUN pip install --no-cache-dir setuptools virtualenv |
There was a problem hiding this comment.
Non-blocking comments
nit:
- Please file a bug with a stack trace that shows how we have a dependency on
distutilsso we can understand whether it can be dropped as reducing the number of dependencies will speed up build times. - Is
virtualenvrequired or can we can recommendvenvto users instead since it's built-in? venv is used in python Client Library READMEs.
There was a problem hiding this comment.
@parthea It looks like any package that requires building a wheel has a stack trace like the one below. It doesn't appear to be actually required by the code within the sample.
We could add distutils (via setuptools) to the requirements-test.txt for each of the current packages with this issue, but we will face unknown errors in the future as packages add dependencies.
Please let me know how you would like to proceed:
Collecting numpy<1.25.0,>=1.14.3 (from apache-beam==2.46.0->apache-beam[gcp]==2.46.0->-r requirements.txt (line 1))
Downloading numpy-1.24.4.tar.gz (10.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.9/10.9 MB 59.9 MB/s eta 0:00:00
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
ERROR: Exception:
Traceback (most recent call last):
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
return func(self, options, args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/commands/install.py", line 377, in run
requirement_set = resolver.resolve(
^^^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
result = self._result = resolver.resolve(
^^^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 427, in resolve
failure_causes = self._attempt_to_pin_criterion(name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 239, in _attempt_to_pin_criterion
criteria = self._get_updated_criteria(candidate)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 230, in _get_updated_criteria
self._add_to_criteria(criteria, requirement, parent=candidate)
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
if not criterion.candidates:
^^^^^^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
return bool(self._sequence)
^^^^^^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
return any(self)
^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
return (c for c in iterator if id(c) not in self._incompatible_ids)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
candidate = func()
^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 182, in _make_candidate_from_link
base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 228, in _make_base_candidate_from_link
self._link_candidate_cache[link] = LinkCandidate(
^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 290, in __init__
super().__init__(
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
self.dist = self._prepare()
^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 222, in _prepare
dist = self._prepare_distribution()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 301, in _prepare_distribution
return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 525, in prepare_linked_requirement
return self._prepare_linked_requirement(req, parallel_builds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 640, in _prepare_linked_requirement
dist = _get_prepared_distribution(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 71, in _get_prepared_distribution
abstract_dist.prepare_distribution_metadata(
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 54, in prepare_distribution_metadata
self._install_build_reqs(finder)
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 124, in _install_build_reqs
build_reqs = self._get_build_requires_wheel()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 101, in _get_build_requires_wheel
return backend.get_requires_for_build_wheel()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_internal/utils/misc.py", line 745, in get_requires_for_build_wheel
return super().get_requires_for_build_wheel(config_settings=cs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
return self._call_hook('get_requires_for_build_wheel', {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 321, in _call_hook
raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
File "/workspace/dataflow/gpu-examples/tensorflow-minimal/.nox/py-3-12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
obj = import_module(mod_path)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
File "<frozen importlib._bootstrap>", line 1304, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 994, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "/tmp/pip-build-env-e4pdkn11/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 10, in <module>
import distutils.core
There was a problem hiding this comment.
I don't think virtualenv is required anymore, but I'd like to test its impact separately. We can try this as a separate PR (and upgrade the Cloud SDK version at the same time).
m-strzelczyk
approved these changes
Mar 25, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Python 3.12 removed distutils (details).
As a result, several 3.12 sample tests are failing. Any time a dependency wheel is built (like
appengine/flexible_python37_and_earlier/scipyhere), we get an error.The release note mentions distutils is still available through setuptools. This PR adds setuptools to the build image.