Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for cleaning up python lib formulae #167905

Open
1 of 8 tasks
cho-m opened this issue Apr 3, 2024 · 10 comments
Open
1 of 8 tasks

Tracking issue for cleaning up python lib formulae #167905

cho-m opened this issue Apr 3, 2024 · 10 comments
Labels
in progress Stale bot should stay away python-vendoring Part of the Python resource vendoring project

Comments

@cho-m
Copy link
Member

cho-m commented Apr 3, 2024

A dedicated tracker for any tasks to handle based on decision in #157500.

For Homebrew/core:

  • Triage python library formulae:
    • Remove python library formulae that do not meet our acceptable requirements and have low number of on-request installs
    • Deprecate python library formulae that do not meet our acceptable requirements and have high number of on-request installs (or have been around for a while in Homebrew/core). We can do a 3mo deprecation + 3mo disable period.
    • Convert/revert Python application formulae that are installed to prefix to instead use virtualenvs
    • Document anything remaining as approved exceptions

Extras for Homebrew/brew:

  • Improve audit checks, particularly for --new formulae, e.g. audit: check for Python-wide site-package usage brew#16663
  • Improve brew doctor and --ignore-dependencies "recommendation" for removed formulae to avoid users blindly running commands
  • Optional: Maybe perform automatic "migration" to new rebuild/revision for removed formulae (which would also need to handle tab of recursive dependents)

Quick table based on 90-day install-on-request analytics data for formulae that may expose libs in HOMEBREW_PREFIX/lib. Also added whether the formula provides any files in bin and non-pure python libs.

Formula 90-day
installs
90-day
rank
# direct
usage
Comment bin/*
cmd?
Ext
libs
pycparser 38,795 136 4 dependency of cryptography
numpy 38,584 138 44 keep, Homebrew/brew#16662 ✅︎ ✅︎
cffi 33,997 154 7 dependency of cryptography ✅︎
mercurial 16,247 273 4 keep ✅︎ ✅︎
pillow 14,684 293 21 keep ✅︎
pyqt@5 10,765 352 5 keep ✅︎ ✅︎
pyqt 10,336 363 3 keep ✅︎ ✅︎
python-setuptools 7,381 463 87
sip 6,932 477 2 ✅︎
cryptography 6,285 511 69 keep, Homebrew/brew#16662 ✅︎
scipy 6,248 512 4 keep, Homebrew/brew#16662 ✅︎
six 2,887 806 4
python-matplotlib 2,823 820 0 ✅︎
python-tabulate 1,771 1083 0 keep #175440 ✅︎
certifi 1,707 1109 164 keep, Homebrew/brew#16662
python-lxml 1,648 1135 0 deprecate, #171823 ✅︎
python-packaging 1,547 1189 1 deprecate #176511
pyyaml 1,319 1293 3 deprecate #176591 ✅︎
python-markdown 1,293 1313 0 keep #175442, decide on rename ✅︎
python-build 1,564 1176 1 reverted to venv, #165713
Decide if rename?
✅︎
pymol 964 1577 0 ✅︎ ✅︎
pygit2 904 1630 0 ✅︎
pymupdf 824 1717 1 ✅︎ ✅︎
python-argcomplete 813 1731 0 ✅︎
lit 678 1908 1 ✅︎
wxpython 639 1962 0 ✅︎ ✅︎
python-ply 559 2101 0 deprecate, #171825
python-chardet 494 2260 0 keep #175446 ✅︎
pyqt-builder 214 3530 3 ✅︎
ly 58 6025 0 keep bin, decide on lib. ✅︎

Table for completed tasks (i.e. merged PRs), excluding removals:

Formula 90-day
installs
90-day
rank
# direct
usage
Decision/PR bin/*
cmd?
Ext
libs
docutils 32,666 162 10 reverted to venv, #168183 ✅︎
python-cryptography 24,100 205 0 renamed to cryptography n/a n/a
python-requests 4,118 650 0 deprecated, #166056
python-certifi 2,262 928 0 renamed to certifi n/a n/a
python-pytz 1,104 1447 0 deprecated, #168071
python-typing-extensions 1,033 1499 1 deprecated, #168114
python-urllib3 877 1667 1 deprecated, #166056
asciidoc 696 1886 25 reverted to venv, #165317 ✅︎
python-dateutil 644 1959 0 deprecated, #168071
python-psutil 603 2019 0 deprecated, #168071 ✅︎
python-trove-classifiers 507 2224 0 deprecated, #168181
python-idna 441 2385 1 deprecated, #166056
python-charset-normalizer 218 3503 1 deprecated, #166056

Collapsed table for removed formulae

Formula 90-day
installs
90-day
rank
# direct
usage
Decision/PR bin/*
cmd?
Ext
libs
python-markupsafe 1,742 1093 0 removed, #165568 n/a n/a
python-platformdirs 447 2369 0 removed, #163603 n/a n/a
python-boto3 398 2511 0 removed, #163229 n/a n/a
python-pyparsing 389 2541 0 removed, #168201
python-jinja 366 2631 0 removed, #165495 n/a n/a
python-hatchling 339 2749 0 removed, #166358 n/a n/a
python-distlib 247 3277 0 removed, #163570
meson-python 235 3363 0 removed, #166060
python-setuptools-scm 179 3879 0 removed, #168073
python-flit-core 174 3942 0 removed, #168201
python-click 171 3983 0 removed, #165496
python-toml 170 4000 0 removed, #165493
python-sympy 169 4010 0 removed, #165494
python-mako 169 4016 0 removed, #165568
python-pluggy 149 4239 0 removed, #166358
python-pycurl 148 4247 0 removed, #164368
python-mutagen 116 4715 0 removed, #164365
python-pathspec 114 4742 0 removed, #166358
python-brotli 107 4869 0 removed, #165112
python-attrs 102 4937 0 removed, #163625
python-networkx 95 5069 0 removed, #166185
python-botocore 84 5306 0 removed, #163229
python-magic 81 5374 0 removed, #164158
python-filelock 78 5446 0 removed, #164940
python-openapi3 71 5638 0 removed, #163442
python-cli-helpers 70 5666 0 removed, #163436
python-colorama 69 5694 0 removed, #163437
python-distro 67 5750 0 removed, #164160
python-cycler 64 5839 0 removed, #166060
python-abseil 62 5908 0 removed, #163343
python-regex 61 5918 0 removed, #164146
python-docopt 55 6100 0 removed, #164149
python-pbr 55 6134 0 removed, #164154
python-configargparse 52 6242 0 removed, #163438
python-hatch-vcs 52 6243 0 removed, #164144
python-json5 52 6244 0 removed, #163642
python-xlsxwriter 51 6288 0 removed, #163646
python-kiwisolver 49 6382 0 removed, #166060 ✅︎
python-requests-oauthlib 48 6421 0 removed, #163590
python-hatch-fancy-pypi-readme 47 6462 0 removed, #164085
python-prompt-toolkit 47 6463 0 removed, #163576
python-termcolor 46 6511 0 removed, #163378
python-rich 45 6557 0 removed, #163623
python-asn1crypto 43 6655 0 removed, #163363
python-oauthlib 43 6656 0 removed, #163712
python-cachetools 41 6745 0 removed, #163425
python-pyproject-hooks 39 6835 0 removed, #166183
python-websocket-client 39 6836 0 removed, #163645
python-dicttoxml 32 7219 0 removed, #163567
python-mpmath 32 7220 0 removed, #165494
python-markdown-it-py 30 7362 0 removed, #164086
python-anytree 29 7431 0 removed, #163424
python-wcwidth 28 7489 0 removed, #163592
python-msgpack 27 7555 0 removed, #164148
python-jmespath 13 9027 0 removed, #163229
python-configobj 7 10540 0 removed, #163565
python-mdurl 7 10541 0 removed, #164086
python-s3transfer 6 10927 0 removed, #163229
@cho-m cho-m added the in progress Stale bot should stay away label Apr 3, 2024
@carlocab
Copy link
Member

carlocab commented Apr 3, 2024

Should probably keep pygit2 -- it links with libgit2, and vendoring it into its dependents leads to really long build times (due to revision bumps) when libgit2 is updated.

@LecrisUT
Copy link
Contributor

I've opened an issue to point out a potential problems with the build process, specifically affecting python projects which have setuptools-scm as a dependency

@offizium-berndstorath
Copy link

What is the reason for removing python-requests?

@SMillerDev
Copy link
Member

What is the reason for removing python-requests?

It's a library, not something that's useful on its own. People are better off adding it to their requirements.txt if they need it somewhere.

@saurik
Copy link

saurik commented Sep 4, 2024

FWIW, my pitch for python-packaging -- apparently now deprecated and soon to be removed due to #176511 -- is that I see it being used as part of peoples' meson builds, and the workaround for it not being part of the distribution is both annoying and non-obvious (I certainly wouldn't have done it correctly), and yet winds up being forced on everyone attempting to compile such packages, whether as a developer automating their build system or as a user who wants to one-off build a random project they checked out which uses meson.

You can actually see this at work in the glib formula: instead of merely marking python-packaging as a build dependency, it instead must vendor a specific version of the package -- which already feels wrong, as this is something that is going to have to get updated, and yet is going to be more coupled to python than glib -- and then futz around with the PYTHON_PATH to inject it into the system python search path, as, apparently, doing this work using a virtual env and passing that separate python context to meson causes glib to not work correctly, per the giant comment in the formula.

resource "packaging" do
url "https://files.pythonhosted.org/packages/51/65/50db4dda066951078f0a96cf12f4b9ada6e4b811516bf0262c0f4f7064d4/packaging-24.1.tar.gz"
sha256 "026ed72c8ed3fcce5bf8950572258698927fd1dbda10a5e981cdf0ac37f4f002"
end

# We don't use a venv as virtualenv_create runs `ENV.refurbish_args`. This causes `gint64`
# to be detected as `long` rather than `long long` on macOS which mismatches with `int64_t`.
# Although documented as valid (https://docs.gtk.org/glib/types.html#gint64), it can cause
# ABI breakage if type changes between releases (e.g. seen in `[email protected]`) and it breaks
# assumptions made by some dependents. Also, GNOME prefers equivalence of types but cannot
# require it due to ABI impact - https://gitlab.gnome.org/GNOME/glib/-/merge_requests/2841
resource("packaging").stage do
system "python3.12", "-m", "pip", "install", "--target", share/"glib-2.0",
*std_pip_args(prefix: false, build_isolation: true), "."
end
ENV.prepend_path "PYTHONPATH", share/"glib-2.0"

I also found a mention by a maintainer of meson (@eli-schwartz) in a discussion of what modules might be useful to include in a standalone copy of meson into which one cannot install additional dependencies--which, FWIW, seems to be the case with Homebrew now, given that python3 is externally managed and these packages that are normally used to accomplish the same goals are largely being deprecated--and packaging is specifically and narrowly called out as "probably a good thing to include" as it is apparently being commonly in various meson build scripts as a replacement for some useful functions from distutils.

mesonbuild/meson#13007 (reply in thread)

The official MSI installer for meson on Windows doesn't include any third-party modules so that probably encourages users to avoid overreliance on such modules. However the packaging module does contain the version parsing and comparison code that replaces distutils functions, so that's probably a good thing to include just in case as people may have ported away from distutils in their script commands.

@cho-m @woodruffw Is there any hope of getting a stay of execution on the python-packaging formula?

@carlocab
Copy link
Member

carlocab commented Sep 4, 2024

Is there any hope of getting a stay of execution on the python-packaging formula?

The discussion above seems like good reasoning for this for me, but CC @Homebrew/core for thoughts.

@MikeMcQuaid
Copy link
Member

You can actually see this at work in the glib formula: instead of merely marking python-packaging as a build dependency, it instead must vendor a specific version of the package -- which already feels wrong, as this is something that is going to have to get updated, and yet is going to be more coupled to python than glib -- and then futz around with the PYTHON_PATH to inject it into the system python search path, as, apparently, doing this work using a virtual env and passing that separate python context to meson causes glib to not work correctly, per the giant comment in the formula.

This seems like a very good reasoning for me along to have python-packaging as a formula.

Really, anything we're repeatedly resourceing in multiple formulae in this sort of non-trivial way feels better handled as a resource.

@SMillerDev
Copy link
Member

Really, anything we're repeatedly resourceing in multiple formulae in this sort of non-trivial way feels better handled as a resource.

You mean resources where we have to mess with pythonpath might be eligible to be formulae right? Not just "anything that we resource a lot" because that used to be the strategy.

@MikeMcQuaid
Copy link
Member

You mean resources where we have to mess with pythonpath might be eligible to be formulae right? Not just "anything that we resource a lot" because that used to be the strategy.

Resources where we're having to write elaborate explanations and provide workarounds like this and we have helpful community members like @saurik taking the time to explain why.

I don't think a "follow this rule and you don't have to think about it" approach is going to work in either direction (not that I'm claiming you're saying that either, to be clear).

carlocab added a commit that referenced this issue Sep 4, 2024
@carlocab
Copy link
Member

carlocab commented Sep 4, 2024

Un-disable!d python-packaging at #183426.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in progress Stale bot should stay away python-vendoring Part of the Python resource vendoring project
Projects
None yet
Development

No branches or pull requests

10 participants