-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Apparent race condition in "waiting for" recently-published package to be available #14644
Comments
@rust-lang/cargo I'm not sure, but this might be on your side? https://github.com/rust-lang/crates.io-index/commits/ca08a581d1282b5ffe1813ebe2da56e4ba83594f looks like the index was updated in the correct order so idk what happened here. this might be a race condition or something like that on the index updates performed by cargo 🤔 |
Yeah, I'm assuming this is on Cargo's side but I have no idea what it could be. The "wait for package" code is basically doing the same check that the resolver is doing when it failed here. We re-download the index until we see the entry. The fact the failed publish is in its own process guarantees there wasn't a stale in-memory cache (compared to our upcoming workspace-publish where its all in one process). What version of Cargo was used for this? |
Is it possible that there's caching happening on the server side of things? Maybe load balancing or something else that'd introduce the possibility of eventual consistency? I'm obviously just guessing since I don't know anything about crates.io's backend design.
|
I would assume the Fastly purge is not atomic. I would guess that the first call hit a server that got purged (or was a cache miss), and the second hit a server that had not yet purged. |
So the assumption is that the "wait for publish" got a result back from an updated server and then the second publish got a stale result, overwriting the new file and causing it to be removed from our on-disk caches? And I assume we use an etag that is only |
Correct, I believe if-none-match is etag equality. |
I guess it's time to /cc @rust-lang/infra since they are in charge of the CDNs and we don't have access to them ourselves. |
FWIW the CDN is sending a |
@rustbot transfer cargo |
This happened at approximately 9:20am Pacific time on 2024-09-27 in case it's useful to cross-reference against logs.
I ran the following commands at google/zerocopy@ec2cfff in order to publish (absolute path prefixes have been redacted). Note that there are two commands being run here (see
cargo publish -p zerocopy
about 2/3 of the way down). I'm including the output verbatim in case that's relevant, but apologies for it being somewhat verbose.If I'm reading this correctly, this line...
...successfully blocked until zerocopy-derive was published, but the subsequent
cargo publish --package zerocopy
command nonetheless observed zerocopy-derive as not yet published. My understanding is that the "waiting for" feature is intended to ensure that this use case (automatically publishing one crate immediately followed by publishing another one) always succeeds without a race condition.The text was updated successfully, but these errors were encountered: