Auto merge of #14662 - epage:resolver, r=Eh2406,weihanglo

docs(resolver): Lay groundwork for documenting MSRV-aware logic

### What does this PR try to resolve?

This is prep for document the MSRV-aware resolver (see #14639), in particular
- This give more breathing room for adding this new heuristic to the resolver documentation
- This provides the context for understanding the limitations

In moving documentation, I asked the question "where would I look to find this if I had a question on it".  I tried to balance this by not putting too much formal / technical documentation in more guide-level descriptions.  In particular, while "Specifying Dependencies" is in the reference, its also written in somewhat of a guide-style.

There is likely more work that can be done, including
- Maybe making the "SemVer Compatibility" chapter the de facto reference for Cargo's version of semver that other sections reference for a more exhaustive description.
- Splitting discussion of the Feature resolver out of the resolver and features documentation.  In the current implementation, we have 3 resolve phases (1) lockfile, (2) adapt to the current compilation, (3) resolve features.  The last two really serve the same role and I'd consider merging discussion of them.

### How should we test and review this PR?

I tried to break up changes in smaller to digest chunks.  This means some times a new section doesn't fully jive with another section until a follow up commit.  I'd recommend reviewing by commit while having the full diff up on the side to see if a concern is still relevant.

### Additional information

src/doc/src/SUMMARY.md

@@ -26,7 +26,7 @@
     * [Specifying Dependencies](reference/specifying-dependencies.md)
         * [Overriding Dependencies](reference/overriding-dependencies.md)
         * [Source Replacement](reference/source-replacement.md)
-    * [Dependency Resolution](reference/resolver.md)
+        * [Dependency Resolution](reference/resolver.md)
     * [Features](reference/features.md)
         * [Features Examples](reference/features-examples.md)
     * [Profiles](reference/profiles.md)

src/doc/src/reference/manifest.md

@@ -92,28 +92,30 @@ a keyword. [crates.io] imposes even more restrictions, such as:
 
 ### The `version` field
 
-Cargo bakes in the concept of [Semantic
-Versioning](https://semver.org/), so make sure you follow some basic rules:
-
-* Before you reach 1.0.0, anything goes, but if you make breaking changes,
-  increment the minor version. In Rust, breaking changes include adding fields to
-  structs or variants to enums.
-* After 1.0.0, only make breaking changes when you increment the major version.
-  Don’t break the build.
-* After 1.0.0, don’t add any new public API (no new `pub` anything) in patch-level
-  versions. Always increment the minor version if you add any new `pub` structs,
-  traits, fields, types, functions, methods or anything else.
-* Use version numbers with three numeric parts such as 1.0.0 rather than 1.0.
+The `version` field is formatted according to the [SemVer] specification:
 
+Versions must have three numeric parts,
+the major version, the minor version, and the patch version.
+
+A pre-release part can be added after a dash such as `1.0.0-alpha`.
+The pre-release part may be separated with periods to distinguish separate
+components. Numeric components will use numeric comparison while
+everything else will be compared lexicographically.
+For example, `1.0.0-alpha.11` is higher than `1.0.0-alpha.4`.
+
+A metadata part can be added after a plus, such as `1.0.0+21AF26D3`.
+This is for informational purposes only and is generally ignored by Cargo.
+
+Cargo bakes in the concept of [Semantic Versioning](https://semver.org/),
+so versions are considered considered [compatible](semver.md) if their left-most non-zero major/minor/patch component is the same.
 See the [Resolver] chapter for more information on how Cargo uses versions to
-resolve dependencies, and for guidelines on setting your own version. See the
-[SemVer compatibility] chapter for more details on exactly what constitutes a
-breaking change.
+resolve dependencies.
 
 This field is optional and defaults to `0.0.0`.  The field is required for publishing packages.
 
 > **MSRV:** Before 1.75, this field was required
 
+[SemVer]: https://semver.org
 [Resolver]: resolver.md
 [SemVer compatibility]: semver.md
 

src/doc/src/reference/resolver.md

@@ -5,99 +5,125 @@ use based on the version requirements specified in each package. This process
 is called "dependency resolution" and is performed by the "resolver". The
 result of the resolution is stored in the `Cargo.lock` file which "locks" the
 dependencies to specific versions, and keeps them fixed over time.
-
-The resolver attempts to unify common dependencies while considering possibly
-conflicting requirements. It turns out, however, that in many cases there is no
-single "best" dependency resolution, and so the resolver must use heuristics to
-choose a preferred solution. The sections below provide some details on how
-requirements are handled, and how to work with the resolver.
-
-See the chapter [Specifying Dependencies] for more details about how
-dependency requirements are specified.
-
 The [`cargo tree`] command can be used to visualize the result of the
 resolver.
 
-[Specifying Dependencies]: specifying-dependencies.md
+[dependency specifications]: specifying-dependencies.md
+[dependency specification]: specifying-dependencies.md
 [`cargo tree`]: ../commands/cargo-tree.md
 
-## SemVer compatibility
-
-Cargo uses [SemVer] for specifying version numbers. This establishes a common
-convention for what is compatible between different versions of a package. See
-the [SemVer Compatibility] chapter for guidance on what is considered a
-"compatible" change. This notion of "compatibility" is important because Cargo
-assumes it should be safe to update a dependency within a compatibility range
-without breaking the build.
-
-Versions are considered compatible if their left-most non-zero
-major/minor/patch component is the same. For example, `1.0.3` and `1.1.0` are
-considered compatible, and thus it should be safe to update from the older
-release to the newer one. However, an update from `1.1.0` to `2.0.0` would not
-be allowed to be made automatically. This convention also applies to versions
-with leading zeros. For example, `0.1.0` and `0.1.2` are compatible, but
-`0.1.0` and `0.2.0` are not. Similarly, `0.0.1` and `0.0.2` are not
-compatible.
-
-As a quick refresher, the
-[*version requirement* syntax][Specifying Dependencies] Cargo uses for
-dependencies is:
-
-Requirement | Example | Equivalence | Description
-------------|---------|-------------|-------------
-Caret | `1.2.3` or `^1.2.3` | <code>>=1.2.3,&nbsp;<2.0.0</code> | Any SemVer-compatible version of at least the given value.
-Tilde | `~1.2` | <code>>=1.2.0,&nbsp;<1.3.0</code> | Minimum version, with restricted compatibility range.
-Wildcard | `1.*` | <code>>=1.0.0,&nbsp;<2.0.0</code> | Any version in the `*` position.
-Equals | `=1.2.3` | <code>=1.2.3</code> | Exactly the specified version only.
-Comparison | `>1.1` | <code>>=1.2.0</code> | Naive numeric comparison of specified digits.
-Compound | <code>>=1.2,&nbsp;<1.5</code> | <code>>=1.2.0,&nbsp;<1.5.0</code> | Multiple requirements that must be simultaneously satisfied.
-
-When multiple packages specify a dependency for a common package, the resolver
-attempts to ensure that they use the same version of that common package, as
-long as they are within a SemVer compatibility range. It also attempts to use
-the greatest version currently available within that compatibility range. For
-example, if there are two packages in the resolve graph with the following
-requirements:
+## Constraints and Heuristics
+
+In many cases there is no single "best" dependency resolution.
+The resolver operates under various constraints and heuristics to find a generally applicable resolution.
+To understand how these interact, it is helpful to have a coarse understanding of how dependency resolution works.
+
+This pseudo-code approximates what Cargo's resolver does:
+```rust
+pub fn resolve(workspace: &[Package], policy: Policy) -> Option<ResolveGraph> {
+    let dep_queue = Queue::new(workspace);
+    let resolved = ResolveGraph::new();
+    resolve_next(pkq_queue, resolved, policy)
+}
+
+fn resolve_next(dep_queue: Queue, resolved: ResolveGraph, policy: Policy) -> Option<ResolveGraph> {
+    let Some(dep_spec) = policy.pick_next_dep(dep_queue) else {
+        // Done
+        return Some(resolved);
+    };
+
+    if let Some(resolved) = policy.try_unify_version(dep_spec, resolved.clone()) {
+        return Some(resolved);
+    }
+
+    let dep_versions = dep_spec.lookup_versions()?;
+    let mut dep_versions = policy.filter_versions(dep_spec, dep_versions);
+    while let Some(dep_version) = policy.pick_next_version(&mut dep_versions) {
+        if policy.needs_version_unification(dep_version, &resolved) {
+            continue;
+        }
+
+        let mut dep_queue = dep_queue.clone();
+        dep_queue.enqueue(dep_version.dependencies);
+        let mut resolved = resolved.clone();
+        resolved.register(dep_version);
+        if let Some(resolved) = resolve_next(dep_queue, resolved) {
+            return Some(resolved);
+        }
+    }
+
+    // No valid solution found, backtrack and `pick_next_version`
+    None
+}
+```
 
+Key steps:
+- Walking dependencies (`pick_next_dep`):
+  The order dependencies are walked can affect
+  how related version requirements for the same dependency get resolved, see unifying versions,
+  and how much the resolver backtracks, affecting resolver performance,
+- Unifying versions (`try_unify_version`, `needs_version_unification`):
+  Cargo reuses versions where possible to reduce build times and allow types from common dependencies to be passed between APIs.
+  If multiple versions would have been unified if it wasn't for conflicts in their [dependency specifications], Cargo will backtrack, erroring if no solution is found, rather than selecting multiple versions.
+  A [dependency specification] or Cargo may decide that a version is undesirable,
+  preferring to backtrack or error rather than use it.
+- Preferring versions (`pick_next_version`):
+  Cargo may decide that it should prefer a specific version,
+  falling back to the next version when backtracking.
+
+### Version numbers
+
+Cargo prefers the highest version currently available.
+
+For example, if you had a package in the resolve graph with:
 ```toml
-# Package A
 [dependencies]
-bitflags = "1.0"
+bitflags = "*"
+```
+If at the time the `Cargo.lock` file is generated, the greatest version of
+`bitflags` is `1.2.1`, then the package will use `1.2.1`.
 
-# Package B
+### Version requirements
+
+Package specify what versions they support, rejecting all others, through
+[version requirements].
+
+For example, if you had a package in the resolve graph with:
+```toml
 [dependencies]
-bitflags = "1.1"
+bitflags = "1.0"  # meaning `>=1.0.0,<2.0.0`
 ```
-
 If at the time the `Cargo.lock` file is generated, the greatest version of
-`bitflags` is `1.2.1`, then both packages will use `1.2.1` because it is the
+`bitflags` is `1.2.1`, then the package will use `1.2.1` because it is the
 greatest within the compatibility range. If `2.0.0` is published, it will
 still use `1.2.1` because `2.0.0` is considered incompatible.
 
-If multiple packages have a common dependency with semver-incompatible
-versions, then Cargo will allow this, but will build two separate copies of
-the dependency. For example:
+[version requirements]: specifying-dependencies.md#version-requirement-syntax
+
+### SemVer compatibility
+
+Cargo assumes packages follow [SemVer] and will unify dependency versions if they are
+[SemVer] compatible according to the [Caret version requirements].
+If two compatible versions cannot be unified because of conflicting version requirements,
+Cargo will error.
 
+See the [SemVer Compatibility] chapter for guidance on what is considered a
+"compatible" change.
+
+Examples:
+
+The following two packages will have their dependencies on `bitflags` unified because any version picked will be compatible with each other.
 ```toml
 # Package A
 [dependencies]
-rand = "0.7"
+bitflags = "1.0"  # meaning `>=1.0.0,<2.0.0`
 
 # Package B
 [dependencies]
-rand = "0.6"
+bitflags = "1.1"  # meaning `>=1.1.0,<2.0.0`
 ```
 
-The above will result in Package A using the greatest `0.7` release (`0.7.3`
-at the time of this writing) and Package B will use the greatest `0.6` release
-(`0.6.5` for example). This can lead to potential problems, see the
-[Version-incompatibility hazards] section for more details.
-
-Multiple versions within the same compatibility range are not allowed and will
-result in a resolver error if it is constrained to two different versions
-within a compatibility range. For example, if there are two packages in the
-resolve graph with the following requirements:
-
+The following packages will error because the version requirements conflict, selecting two distinct compatible versions.
 ```toml
 # Package A
 [dependencies]
@@ -108,14 +134,39 @@ log = "=0.4.11"
 log = "=0.4.8"
 ```
 
-The above will fail because it is not allowed to have two separate copies of
-the `0.4` release of the `log` package.
+The following two packages will not have their dependencies on `rand` unified because only incompatible versions are available for each.
+Instead, two different versions (e.g. 0.6.5 and 0.7.3) will be resolved and built.
+This can lead to potential problems, see the [Version-incompatibility hazards] section for more details.
+```toml
+# Package A
+[dependencies]
+rand = "0.7"  # meaning `>=0.7.0,<0.8.0`
+
+# Package B
+[dependencies]
+rand = "0.6"  # meaning `>=0.6.0,<0.7.0`
+```
+
+Generally, the following two packages will not have their dependencies unified because incompatible versions are available that satisfy the version requirements:
+Instead, two different versions (e.g. 0.6.5 and 0.7.3) will be resolved and built.
+The application of other constraints or heuristics may cause these to be unified,
+picking one version (e.g. 0.6.5).
+```toml
+# Package A
+[dependencies]
+rand = ">=0.6,<0.8.0"
+
+# Package B
+[dependencies]
+rand = "0.6"  # meaning `>=0.6.0,<0.7.0`
+```
 
 [SemVer]: https://semver.org/
 [SemVer Compatibility]: semver.md
+[Caret version requirements]: specifying-dependencies.md#default-requirements
 [Version-incompatibility hazards]: #version-incompatibility-hazards
 
-### Version-incompatibility hazards
+#### Version-incompatibility hazards
 
 When multiple versions of a crate appear in the resolve graph, this can cause
 problems when types from those crates are exposed by the crates using them.
@@ -150,54 +201,6 @@ ecosystem if you publish a SemVer-incompatible version of a popular library.
 [semver trick]: https://github.com/dtolnay/semver-trick
 [`downcast_ref`]: ../../std/any/trait.Any.html#method.downcast_ref
 
-### Pre-releases
-
-SemVer has the concept of "pre-releases" with a dash in the version, such as
-`1.0.0-alpha`, or `1.0.0-beta`. Cargo will avoid automatically using
-pre-releases unless explicitly asked. For example, if `1.0.0-alpha` of package
-`foo` is published, then a requirement of `foo = "1.0"` will *not* match, and
-will return an error. The pre-release must be specified, such as `foo =
-"1.0.0-alpha"`. Similarly [`cargo install`] will avoid pre-releases unless
-explicitly asked to install one.
-
-Cargo allows "newer" pre-releases to be used automatically. For example, if
-`1.0.0-beta` is published, then a requirement `foo = "1.0.0-alpha"` will allow
-updating to the `beta` version. Note that this only works on the same release
-version, `foo = "1.0.0-alpha"` will not allow updating to `foo = "1.0.1-alpha"`
-or `foo = "1.0.1-beta"`.
-
-Cargo will also upgrade automatically to semver-compatible released versions
-from prereleases. The requirement `foo = "1.0.0-alpha"` will allow updating to
-`foo = "1.0.0"` as well as `foo = "1.2.0"`.
-
-Beware that pre-release versions can be unstable, and as such care should be
-taken when using them. Some projects may choose to publish breaking changes
-between pre-release versions. It is recommended to not use pre-release
-dependencies in a library if your library is not also a pre-release. Care
-should also be taken when updating your `Cargo.lock`, and be prepared if a
-pre-release update causes issues.
-
-The pre-release tag may be separated with periods to distinguish separate
-components. Numeric components will use numeric comparison. For example,
-`1.0.0-alpha.4` will use numeric comparison for the `4` component. That means
-that if `1.0.0-alpha.11` is published, that will be chosen as the greatest
-release. Non-numeric components are compared lexicographically.
-
-[`cargo install`]: ../commands/cargo-install.md
-
-### Version metadata
-
-SemVer has the concept of "version metadata" with a plus in the version, such
-as `1.0.0+21AF26D3`. This metadata is usually ignored, and should not be used
-in a version requirement. You should never publish multiple versions that
-differ only in the metadata tag.
-
-## Other constraints
-
-Version requirements aren't the only constraint that the resolver considers
-when selecting and unifying dependencies. The following sections cover some of
-the other constraints that can affect resolution.
-
 ### Features
 
 For the purpose of generating `Cargo.lock`, the resolver builds the dependency
@@ -610,3 +613,20 @@ circumstances:
   change of your own library, for example if it exposes types from the
   dependency.
 
+[`cargo install`]: ../commands/cargo-install.md
+
+<script>
+(function() {
+    var fragments = {
+        "#version-metadata": "specifying-dependencies.html#version-metadata",
+        "#pre-releases": "specifying-dependencies.html#pre-releases",
+        "#other-constraints": "#constraints-and-heuristics",
+    };
+    var target = fragments[window.location.hash];
+    if (target) {
+        var url = window.location.toString();
+        var base = url.substring(0, url.lastIndexOf('/'));
+        window.location.replace(base + "/" + target);
+    }
+})();
+</script>