pixi panics when built with Rust 1.81

[mdekstrand]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

pixi.toml / pyproject.toml: in lenskit/lkpy#483

pixi update

When running Pixi built with Rust 1.81.0, this panics. The official binaries and Rust 1.80.0 both succeed.

error output with RUST_BACKTRACE=1

note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'tokio-runtime-worker' panicked at library/core/src/slice/sort/shared/smallsort.rs:862:5:
user-provided comparison function does not correctly implement a total order
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::slice::sort::shared::smallsort::panic_on_ord_violation
   3: core::slice::sort::shared::smallsort::small_sort_general_with_scratch
   4: core::slice::sort::stable::quicksort::quicksort
   5: core::slice::sort::stable::quicksort::quicksort
   6: core::slice::sort::stable::quicksort::quicksort
   7: core::slice::sort::stable::quicksort::quicksort
   8: core::slice::sort::stable::quicksort::quicksort
   9: core::slice::sort::stable::drift::sort
  10: core::slice::sort::stable::driftsort_main
  11: resolvo::solver::cache::SolverCache<D>::get_or_cache_sorted_candidates_for_version_set::{{closure}}
  12: resolvo::solver::cache::SolverCache<D>::get_or_cache_sorted_candidates::{{closure}}
  13: resolvo::solver::Solver<D,RT>::add_clauses_for_solvables::{{closure}}::{{closure}}
  14: futures_util::stream::stream::StreamExt::poll_next_unpin
  15: futures_util::future::future::FutureExt::now_or_never
  16: resolvo::solver::Solver<D,RT>::run_sat
  17: resolvo::solver::Solver<D,RT>::solve
  18: <rattler_solve::resolvo::Solver as rattler_solve::SolverImpl>::solve
  19: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  20: tokio::runtime::task::core::Core<T,S>::poll
  21: tokio::runtime::task::harness::Harness<T,S>::poll
  22: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'tokio-runtime-worker' panicked at library/core/src/slice/sort/shared/smallsort.rs:862:5:
user-provided comparison function does not correctly implement a total order
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::slice::sort::shared::smallsort::panic_on_ord_violation
   3: core::slice::sort::shared::smallsort::small_sort_general_with_scratch
   4: core::slice::sort::stable::quicksort::quicksort
   5: core::slice::sort::stable::quicksort::quicksort
   6: core::slice::sort::stable::quicksort::quicksort
   7: core::slice::sort::stable::quicksort::quicksort
   8: core::slice::sort::stable::quicksort::quicksort
   9: core::slice::sort::stable::drift::sort
  10: core::slice::sort::stable::driftsort_main
  11: resolvo::solver::cache::SolverCache<D>::get_or_cache_sorted_candidates_for_version_set::{{closure}}
  12: resolvo::solver::cache::SolverCache<D>::get_or_cache_sorted_candidates::{{closure}}
  13: resolvo::solver::Solver<D,RT>::add_clauses_for_solvables::{{closure}}::{{closure}}
  14: futures_util::stream::stream::StreamExt::poll_next_unpin
  15: futures_util::future::future::FutureExt::now_or_never
  16: resolvo::solver::Solver<D,RT>::run_sat
  17: resolvo::solver::Solver<D,RT>::solve
  18: <rattler_solve::resolvo::Solver as rattler_solve::SolverImpl>::solve
  19: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  20: tokio::runtime::task::core::Core<T,S>::poll
  21: tokio::runtime::task::harness::Harness<T,S>::poll
  22: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'tokio-runtime-worker' panicked at library/core/src/slice/sort/shared/smallsort.rs:862:5:
user-provided comparison function does not correctly implement a total order
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::slice::sort::shared::smallsort::panic_on_ord_violation
   3: core::slice::sort::shared::smallsort::small_sort_general_with_scratch
   4: core::slice::sort::stable::quicksort::quicksort
   5: core::slice::sort::stable::quicksort::quicksort
   6: core::slice::sort::stable::quicksort::quicksort
   7: core::slice::sort::stable::quicksort::quicksort
   8: core::slice::sort::stable::quicksort::quicksort
   9: core::slice::sort::stable::drift::sort
  10: core::slice::sort::stable::driftsort_main
  11: resolvo::solver::cache::SolverCache<D>::get_or_cache_sorted_candidates_for_version_set::{{closure}}
  12: resolvo::solver::cache::SolverCache<D>::get_or_cache_sorted_candidates::{{closure}}
  13: resolvo::solver::Solver<D,RT>::add_clauses_for_solvables::{{closure}}::{{closure}}

Issue description

Something in Pixi's code or dependencies (I suspect Rattler or a dependency) appears to be incompatible with some change introduces in Rust 1.81.0. With the same project, pixi update reliably panics when built with 1.81.0, and succeeds when built with 1.80.0 (or using the v0.30.0 official binary, which I presume is built with 1.80.0 since that is in the rust-toolchain file).

This is an observable issue because Homebrew ignores rust-toolchain and builds Pixi with Rust 1.81.0. I see the same panic with brew-installed Pixi on both MacOS and Linux.

I have raised the concern of ignoring rust-toolchain in the Homebrew discussion forums (https://github.com/orgs/Homebrew/discussions/5629), but am also filing a bug here since I expect you all will hit this when you bump your Rust version, and so you are aware of the problem now.

Another, much simpler environment I have correctly resolves with the brew-installed Pixi.

Expected behavior

Environment correctly resolves without panic.

[fpaniagua]

Maybe related:
oxc-project/oxc#5773 (comment)

I get a similar or same issue on pixi via homebrew on a mac. Similar stacktrace.

One thing I noticed in a particular repository, is that it is references conda-forge and main, and I get a lot of output like:

...
DEBUG rattler_solve::resolvo: Ignoring 'libstdcxx-devel_linux-64' from 'https://conda.anaconda.org/conda-forge/' because of strict channel priority.
DEBUG rattler_solve::resolvo: Ignoring 'libstdcxx-devel_linux-64' from 'https://conda.anaconda.org/conda-forge/' because of strict channel priority.
thread 'tokio-runtime-worker' panicked at library/core/src/slice/sort/shared/smallsort.rs:862:5:
user-provided comparison function does not correctly implement a total order

Removing main, causes pixi update to complete.

Seems to also happen with conda-forge and pytorch channels.

[fpaniagua]

This pyproject.yaml appears to reproduce:

# pyproject.toml
[project]
name = "pixi-example"
requires-python = ">=3.9"

[tool.pixi.project]
name = "pixi-example"
channels = ["conda-forge", "pytorch"]
platforms = ["linux-64"]

[tool.pixi.tasks]

[tool.pixi.feature.prod.dependencies]
python = "3.9.*"
pytorch = { version = "1.10.*", channel= "pytorch"}

[tool.pixi.environments]
default = {features = ["prod"], solve-group = "default"}

on pixi v0.30.0 installed from homebrew.

I also installed pixi directly from github(also v0.30.0), and that version like above has no problem with the above pyproject.toml

[ruben-arts]

Thanks for the information, I assume this issue is coming from https://github.com/mamba-org/resolvo or https://github.com/conda/rattler. Which we use in pixi. Is there another reason we should update.

[jonashaag]

I can reproduce this with the Homebrew binary

[mdekstrand]

I went and ran tests and the resolvo tests pass on Rust 1.81.0, but the rattler tests fail (conda/rattler#888). I don't think this definitively isolates the problem to Rattler, since it could be in resolvo with a code path not exercised by tests, but perhaps gets a step closer to isolating and fixing.

[wolfv]

I think it might be an issue in the DependencyProvider which is the part that's responsible for sorting the packages. I'll take a quick look.

[tdejager]

I've a PR in rattler that when merged and pixi is updated should fix this: conda/rattler#892

[mdekstrand]

@tdejager I have tried a local build of Pixi (main branch) with its rattler_solve dependency updated to the version with your PR and using Rust 1.81, and I can confirm that it fixes the sort-related Pixi panic on at least one of my projects (this project panics when main is built with 1.81.0, and does not when built with the patched rattler dependency).

[tdejager]

@tdejager I have tried a local build of Pixi (main branch) with its rattler_solve dependency updated to the version with your PR and using Rust 1.81, and I can confirm that it fixes the sort-related Pixi panic on at least one of my projects (this project panics when main is built with 1.81.0, and does not when built with the patched rattler dependency).

Great 👍 we'll try to push a release in a day or so.

[wolfv]

Release is out :)