-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upgrade setuptools to a more recent version #949
Comments
Also needed for cairocffi: https://github.com/Kozea/cairocffi/blob/master/setup.cfg#L43 Made a PR for this: #973 |
it would be good to get a response to this, or the PR from @rvanlaar with a potential fix, it's been a number of weeks with no acknowledgement |
@russwinch Hi! Sorry for the delayed reply, we've just had a changeover of Python language owner, and it's going to take a while for me to get through the discovery/switching role onboarding process. I very much want to upgrade setuptools and other out of date dependencies in this buildpack, though they are high risk so will need some investigation/time to reduce chance of breaking other customers. |
About breaking other builds, yes it might break their builds, however it currently also breaks builds and prevents users from installing certain packages. |
Just starting my heroku app and installing it gives me same errors, including the cairocffi one.
|
I had the same issue with |
The versions installed by the buildpack have been updated as follows: * pip: - If using Python 3.4: No change (already using the last to support 3.4) - If using pipenv: No change (need to update to a newer pipenv first) - For everything else: `20.0.2` -> `20.1.1` * setuptools: - If using Python 3.4: `39.0.1` -> `43.0.0` (latest for 3.4) - If using Python 2.7: `39.0.1` -> `44.1.1` (latest for 2.7) - For everything else: `39.0.1` -> `47.1.1` (until #1006 fixed) * wheel: - If using Python 3.4: `unpinned` -> `0.33.6` - For everything else: `unpinned` -> `0.34.2` This fixes #949 and fixes #1005, and means packages that rely on newer setuptools will now install successfully. Changelogs: https://pip.pypa.io/en/stable/news/ https://setuptools.readthedocs.io/en/latest/history.html#v47-1-1 https://wheel.readthedocs.io/en/latest/news.html In addition: * Installed versions are now deterministic (fixes #1000, fixes #1003) * The build output now includes the versions used, making it easier to debug future upgrades (closes #939) * Errors during pip/setuptools/wheel install now correctly fail the build, and stderr is no longer sent to `/dev/null` (fixes #1002) * Setuptools is no longer installed twice (fixes #1001) * Everything that is downloaded is now used (fixes #999) * `--no-cache` and `--disable-version-check` are now used, saving unnecessary work and preventing creation of unwanted files in `/app` * The `PIP_UPDATE` env var no longer leaks into subprocesses. As part of fixing version pinning, we now use pip itself to determine whether the installed packages are up to date, since parsing pip's output is fragile (eg #1003). This means `pip install` is now called every time, however this is a no-op for repeat builds where the versions have not changed, since unless `--upgrade` is specified pip does not hit the index (PyPI) if requirements are satisfied. For the installation itself `get-pip.py` is no longer used, since: - It uses `--force-reinstall`, which is unnecessary here and would slow down repeat builds (given we call pip install every time now). Trying to work around this by using `get-pip.py` only for the initial install, and real pip for subsequent updates would mean we lose protection against cached broken installs, plus significantly increase the version combinations test matrix. - It means downloading pip twice (once embedded in `get-pip.py`, and again during the install, since `get-pip.py` can't install the embedded version directly) - We would still have to manage several versions of get-pip.py, to support older Pythons. We don't use `ensurepip` since: - Not all of the previously generated Python runtimes on S3 include it - We would still have to upgrade pip afterwards - The versions of pip/setuptools bundled with ensurepip differ greatly depending on Python version, and we could easily start using a CLI flag for the first pip install before upgrade that isn't supported on all versions, without even knowing it (unless we test against hundreds of Python archives). The new pip wheel assets on S3 were generated using: ``` $ pip download --no-cache pip==19.1.1 Collecting pip==19.1.1 Downloading pip-19.1.1-py2.py3-none-any.whl (1.4 MB) Saved ./pip-19.1.1-py2.py3-none-any.whl Successfully downloaded pip $ pip download --no-cache pip==20.1.1 Collecting pip==20.1.1 Downloading pip-20.1.1-py2.py3-none-any.whl (1.5 MB) Saved ./pip-20.1.1-py2.py3-none-any.whl Successfully downloaded pip $ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read --dryrun (dryrun) upload: ./pip-19.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-19.1.1-py2.py3-none-any.whl (dryrun) upload: ./pip-20.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-20.1.1-py2.py3-none-any.whl $ aws s3 sync . s3://lang-python/common/ --exclude "*" --include "*.whl" --acl public-read upload: ./pip-19.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-19.1.1-py2.py3-none-any.whl upload: ./pip-20.1.1-py2.py3-none-any.whl to s3://lang-python/common/pip-20.1.1-py2.py3-none-any.whl ```
@russwinch @rvanlaar @trevorphillips @jualopver Hi :-) Now that the pip/setuptools/wheel installation steps have been fixed/refactored in #1007, we're ready to upgrade setuptools to a newer release. I was hoping to add a testcase at the same time, to prove that previously broken packages install fine. However I'm unable to reproduce? What version of Python are you using? Are you using Pip or pipenv? What version of the Python buildpack? And can you reproduce on a clean app? For example I tried:
|
Upgrades setuptools from 39.0.1 to: - 44.1.1 for Python 2.7 (since it's the last supported version) - 43.0.0 for Python 3.4 (since it's the last supported version) - 47.1.1 for Python 3.5+ (since we can't use 47.2.0+ until #1006 fixed) https://setuptools.readthedocs.io/en/latest/history.html#v47-1-1 Fixes #949. Closes #973.
Hi @edmorley thanks for having a look at this. I have replicated your results with a clean app
Curiously, this automatic upgrade does not happen when deploying the app I originally saw the error in:
Presumably the issue could be to do with our combination of requirements? But I have upgraded every package that depends on setuptools and have not been able to trigger the setuptools upgrade seen in the clean app. This this app is deployed via the github integration, vs heroku command line in the clean app, if this helps. What version of Python are you using? 3.8.3 |
@russwinch Thank you for that package list - using that I could reproduce. Running
Which led to the reduced testcase of:
I believe the reason pip doesn't upgrade setuptools in this case is pypa/pip/issues/988. ie: For transitive dependencies the first version specified "wins" - and since Thankfully a new pip resolver which fixes this is already in progress: Anyway for now I'll update setuptools which will prevent this being an issue regardless of the resolver :-) |
Upgrades setuptools from 39.0.1 to: - 44.1.1 for Python 2.7 (since it's the last supported version) - 43.0.0 for Python 3.4 (since it's the last supported version) - 47.1.1 for Python 3.5+ (since we can't use 47.2.0+ until #1006 fixed) https://setuptools.readthedocs.io/en/latest/history.html#v47-1-1 Fixes #949. Closes #973.
Upgrades setuptools from 39.0.1 to: - 44.1.1 for Python 2.7 (since it's the last supported version) - 43.0.0 for Python 3.4 (since it's the last supported version) - 47.1.1 for Python 3.5+ (since we can't use 47.2.0+ until #1006 fixed) https://setuptools.readthedocs.io/en/latest/history.html#v47-1-1 Fixes #949. Closes #973.
The version of setuptools has been upgraded in #1024, which has been merged to |
The updated setuptools (and pip) is available in the stable buildpack release (the |
Upgrades setuptools from 39.0.1 to: - 44.1.1 for Python 2.7 (since it's the last supported version) - 43.0.0 for Python 3.4 (since it's the last supported version) - 47.1.1 for Python 3.5+ (since we can't use 47.2.0+ until heroku#1006 fixed) https://setuptools.readthedocs.io/en/latest/history.html#v47-1-1 Fixes heroku#949. Closes heroku#973.
I'm starting to get incompatibility warnings about the version of setuptools supplied in this buildpack:
If I manually
pip install
the google-auth package, setuptools is automatically updated but this doesn't seem the case when the app is built.The version provided in the build pack was released 18 Mar 2018. Latest is 46.1.3
Would it be possible to get an update to setuptools in the next release?
The text was updated successfully, but these errors were encountered: