overrides: integrate poetry2nix_override_bruteforcer (many overrides...)

[TyberiusPrime]

poetry2nix_override_bruteforcer is an attempt to enlarge the set of packages poetry2nix can actually build.

It starts with a 16kish set of python packages (top 8k by downloads, everything with 5 or more pypi dependencies), tries to build them, and then auto-writes build-systems / overrides, downloads cargo.locks etc., all depending on the failure messages.

With a little bit of polishing, these are then ready to integrate into poetry2nix, hence this PR.

I have chosen to integrate them as separate files, since it's so much generated code, and I don't want to pollute the current set of hand-crafted overrides. Also it should make updates later on easy, since we don't need to actually merge files.

And I fully expect that the generated get both larger, and more polished in the future as the scope and sophistication of poetry2nix_override_bruteforcer grows.

Statistics:
The package set is 16375 packages larges (17336 minus packages that I have tagged 'won't ever build).
The base rev I used (which incoporates #1738) builds 6467 of them, or 39.49%.
The auto-generated overrides manage to build another 7667, leading to a 86.31% build rate.

(the actual merged set might build a few fewer. There's some need to merge multiple overrides that I haven't been able to get to yet, and so they're not included here. Same for 'needs different overrides depending on package version'. But it's a small set)

• Commit messages are conventional - examples from the log include "feat: add changelog files to fixup hook", "fix(contourpy): allow wheel usage", and "test: add sqlalchemy2 test"

[Atry]

Some existing tests are broken. Do you know why?

[TyberiusPrime]

Some existing tests are broken. Do you know why?

I haven't had time to investigate, but I expect it to be python package drift - i.e. older versions needing different overrides than the current ones.

Plus maybe the override issue in #1737 .

I'll get it into mergeable state next week, hopefully, had to take a brief break.

[TyberiusPrime]

Once more, with feeling.
It was indeed mostly drift in what the python packages actually need.

I also fixed a bunch of packages that need setuptools_scm or such not to produce 0.0.0 'versions' in the output.

And bunch more epi-circles in the generator.
Let's hope it passes the CI now.

[TyberiusPrime]

Ok, I think the vllm one and the mac one are temporary issues talking with pypi,
for at least vllm builds locally (with a cached xformers wheel).

2024-08-22T08:32:53.4011699Z error: builder for '/nix/store/5885nhv9qmnn165cpza4p4mnj1c2av49-xformers-0.0.26.post1-cp311-cp311-manylinux2014_x86_64.whl.drv' failed with exit code 92;
2024-08-22T08:32:53.4013180Z        last 7 log lines:
2024-08-22T08:32:53.4014748Z        > Trying to fetch with predicted URL: https://files.pythonhosted.org/packages/wheel/x/xformers/xformers-0.0.26.post1-cp311-cp311-manylinux2014_x86_64.whl
2024-08-22T08:32:53.4016224Z        > curl: (22) The requested URL returned error: 404
2024-08-22T08:32:53.4016807Z        >
2024-08-22T08:32:53.4018306Z        > Predicted URL 'https://files.pythonhosted.org/packages/wheel/x/xformers/xformers-0.0.26.post1-cp311-cp311-manylinux2014_x86_64.whl' failed, querying pypi.org
2024-08-22T08:32:53.4020098Z        > ######################################################################## 100.0%
2024-08-22T08:32:53.4021046Z        > ###################################################                       71.5%curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
2024-08-22T08:32:53.4021841Z        >
2024-08-22T08:32:53.4023138Z        For full logs, run 'nix log /nix/store/5885nhv9qmnn165cpza4p4mnj1c2av49-xformers-0.0.26.post1-cp311-cp311-manylinux2014_x86_64.whl.drv'.
2024-08-22T08:32:53.4291648Z building '/nix/store/i4pf4x95q2rjklaxmrr7mci9njmwkzqn-yarl-1.9.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.drv'...

Otherwise, I think I got it in mergeable shape for now.

Obviously, the overrides could use some nix-level refactoring, but I'll have to rethink the architecture of my 'bruteforcer' first.

I needed to do a lot of 'sweeping' of every package version,
so the stats read

Total pkg * versions attempted: 41847
builds without patch: 31.69%
builds after patch 84.22 %

I'll retrigger the CI some time later today, hopefully it the pypi issue will have passed by then.
(edit: I have retriggered the CI. I can nix-store --realise --check the very FOD derivation that failed in CI).

[TyberiusPrime]

Ok, the pypi issues went away but I broke two packages on mac.
Wish I could test that locally, but a quick grep suggests there shouldn't be any more. (fingers crossed...)

[TyberiusPrime]

Great, now the macos runner is running out of disk space ?!

[Atry]

@adisbladis This PR is really useful. How can we get it merged?

[TyberiusPrime]

If you like this PR, you might also like https://github.com/TyberiusPrime/uv2nix_hammer_overrides which is an independent override collection.

[TyberiusPrime]

abandoning this for uv2nix.