Add --extracted flag to uv cache prune to remove unzipped wheels
#5282
Comments
|
The uv cache contains all the wheels we install unpacked (in the |
Same package, wildly different result after running |
|
I think you instead want |
You seem to be right; |
|
I think we should consider adding some options to It doesn't make a ton of sense to be storing |
|
If you see improvement with something like |
|
Interestingly enough, when I execute; I get I kinda expected less matching artifact files then when I just ran I'll see if I can do some performance tuning on our Github helpers CPU & Memory. They seem to be capping. I'll report back afterwards :) |
|
I think "more" is correct. With |
|
Although we cleaned quite a bit, the size is still very large: Using the FF_USE_FASTZIP flag in combination with We have 200MB/s internet speed for our S3 cache back-end, but still it seems faster to just download the artifacts from our Python repository. |
|
So I listed the 100 largest file in the It contains the large Wheel and then it also contains the extracted items. The 90 biggest files are all Jars from this specific wheel. Is there a way to remove these JARs / extracted files? |
|
You could run |
Wow.. It seems that was 858.0MB of the 1.2G, as we have 311MB remaining -> 73% was taken from one package. I think |
|
Like, store the built wheel, but none of the unzipped archives? We could probably support that. |
Exactly that. At that point, caching the |
|
It would be easy to support We could also support unzipping the wheel directly into the virtual environment (rather than to the cache first). That's probably the "right" solution for this use-case, but it's a lot more work. (For pre-built wheels that we download from a registry, we don't even save the wheel itself -- we unzip the extracted files directly to disk.) |
I think the first part would fit my use-case perfectly. At first I was thinking of an environment variable like |
|
That should be straightforward to implement. I'll try to find time, but will also mark this as help wanted. Basically, we want to remove these symlink directories in the
|
charliermarsh
changed the title
--extracted flag to uv cache prune to remove unzipped wheels
charliermarsh
added
enhancement
|
Added here: #5391 |
## Summary Users can now run `uv cache prune --ci` (open to feedback on the name of that flag) to remove all pre-built wheels from the cache, leaving behind zipped, built wheels (which tend to be the most expensive assets to re-create). This should greatly increase cache performance in CI environments, since uploading unzipped wheels can actually hurt performance if you're persisting the uv cache. Closes #5282.
First of all, thank you for the awesome package. It's working blazingly fast and our developers are loving the new speed they have locally.
In our pipelines, the speed has not really improved. The main reason is that pip tools cache is considerably smaller than uv. When we store the uv cache, the Gitlab pipeline is spending much more time on zipping the small files, then it is spending on downloading the files new. The difference is considerable; it's 2 times slower when we use Gitlab caches on the uv cache.
A small app easily generates 16000 small files in uv cache and easily comes to
This is not only visible in Gitlab, but also in Github as mentioned in this ticket;
actions/setup-python#822 (comment)
Would it maybe be possible to add a flag for CI/CD to concentrate the cache in larger files?
The text was updated successfully, but these errors were encountered: