provide a way to not download tests/, examples and other such common directories

[poliorcetics]

Problem

By default cargo includes everything from the root of the package (or almost, see https://doc.rust-lang.org/cargo/reference/manifest.html?highlight=files#the-exclude-and-include-fields)

This is very nice for archiving purposes by crates.io, it allows example scraping from docs.rs and probably other purposes I haven't encountered, but it has an issue: it makes lots of CI downloads heavier than they need to be.

Looking through the dependencies we use at $work I see around a 100MiB of unused files just counting tests/, but there are also .github, benches/, examples/ and probably more I missed, and we have only ~600 deps out of the 138k crates on crates.io (without even counting all the available versions).

We have a cache to avoid spamming crates.io whenever possible so it's not like we request an extra 100MiB on each CI but still, I don't like putting pressure on it if we can avoid it.

As linked above, https://doc.rust-lang.org/cargo/reference/manifest.html?highlight=files#the-exclude-and-include-fields is intended to help with that but it's not automatic: people need to consider and maintain it for each crate (and from the start, else older versions will still have all the unused files)

Proposed Solution

We may need a way to say to cargo only fetch the absolute necessary to build the crate. Two issues with that:

• What is the "absolute necessary": probably at least src/, Cargo.toml, license files, build.rs ? Some -sys crates may need more
• Changing that silently in a backward compatible way is probably hard ? I think it would break the hashes so it could not become the default automatically at the very least.

It could take the form of cargo fetch --no-extras (to be bikeshedded) ?

Notes

Note: I'm excluding binary crates here, but they could easily be included too, if only so cargo install downloads less if possible, but I don't think binary crates are downloaded as much as library ones.

I also don't know if the benefits would be worth it, since it could mean one of:

• Recomputing the set of files on each download
• Storing two archives of the same crate&version

I have no data saying the bandwidth gains would be enough to offset either the compute or space gains, if anyone knows (the infra team ?) I would be happy to be proven wrong!

[epage]

For cargo vendor, we also have #13474.

Breaking down the extraneous files you mentioned:

• .github there is no reason to include and that is on the package for including it.
• For examples/, that can tie into the documentation and can be important for including (this isn't just for crates.io but for local docs builds)
• For tests/ and benches/, that is important for crater and people wanting to do similar for crater.

The challenge is knowing what is needed for build or not among the files. A heuristic can work but that can only go so far (much like we provide package.include and package.exclude in case our existing heuristic isn't enough.

There is also the question of "what is essential". For example, explicitly referenced targets are sometimes required just to parse manifests today (see #13456).

There would need to be figured out to get this to work.

[epage]

For the most part, we are talking text here. I wonder if we'd be better spending our time focusing on aspects of this, like

• Migrate to alternative compression for crates.io crates #2526
• Warning when large binary files are included into the bundle #9058

[weihanglo]

I would like to call out that the #[path] attribute, while not widely used, would cause problems if Cargo excludes files without recognizing where they point to. Besides, changing or disable the heuristics might affect other parts in Carog, like #4587, though this doesn't mean that Cargo cannot change the current rule.

[poliorcetics]

For cargo vendor, we also have #13474.

Ah I looked for cargo fetch issues, I missed that one. It's not exactly the same problem but very close

The challenge is knowing what is needed for build or not among the files. A heuristic can work but that can only go so far (much like we provide package.include and package.exclude in case our existing heuristic isn't enough.

package.include and .exclude must be set by users and I think at this point experience has proven it's not gonna happen often.

I also agree that crates.io should have everything, tests and examples included, I'm not advocating for dropping that! I would like the ability to not download them but as I said, it's hard to know what to download and what not to.

Does cargo check hashes again once a library has been fetched ? If not if would allow me to just delete the unneeded directories and files on my side without interfering with cargo itself

[epage]

I don't think we re-check hashes but i would not rely on that never changing.

[poliorcetics]

i would not rely on that never changing.

It should be a line or two of bash, it can be deleted if an update breaks it :)

[poliorcetics]

Update: cargo is too well-made and check hashes even in vendored dependencies so my two lines of bash won't work 😞

[poliorcetics]

A simpler way to do this then would be to allow not checking hashes when compiling ?