CUDA_ERROR_ILLEGAL_ADDRESS when running Llama3 and Llama3.1

[ShelbyJenkins]

This occurs when using two GPUs, but it does not occur when I use just the one.

I made sure to update to the docker image used in the dockerfile.

commit: a702c6d (from earlier this week)

thread '<unnamed>' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cudarc-0.12.1/src/driver/safe/core.rs:252:76:
called `Result::unwrap()` on an `Err` value: DriverError(CUDA_ERROR_ILLEGAL_ADDRESS, "an illegal memory access was encountered")
stack backtrace:
   0: rust_begin_unwind
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/result.rs:1654:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/result.rs:1077:23
   4: <cudarc::driver::safe::core::CudaSlice<T> as core::ops::drop::Drop>::drop
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cudarc-0.12.1/src/driver/safe/core.rs:252:17
   5: core::ptr::drop_in_place<cudarc::driver::safe::core::CudaSlice<f32>>
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
   6: core::ptr::drop_in_place<candle_core::cuda_backend::CudaStorageSlice>
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
   7: core::ptr::drop_in_place<candle_core::cuda_backend::CudaStorage>
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
   8: core::ptr::drop_in_place<candle_core::storage::Storage>
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
   9: core::ptr::drop_in_place<core::cell::UnsafeCell<candle_core::storage::Storage>>
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
  10: core::ptr::drop_in_place<std::sync::rwlock::RwLock<candle_core::storage::Storage>>
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
  11: alloc::sync::Arc<T,A>::drop_slow
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/alloc/src/sync.rs:1804:18
  12: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/alloc/src/sync.rs:2462:13
  13: core::ptr::drop_in_place<alloc::sync::Arc<std::sync::rwlock::RwLock<candle_core::storage::Storage>>>
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
  14: core::ptr::drop_in_place<candle_core::tensor::Tensor_>
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
  15: alloc::sync::Arc<T,A>::drop_slow
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/alloc/src/sync.rs:1804:18
  16: <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/alloc/src/sync.rs:2462:13
  17: core::ptr::drop_in_place<alloc::sync::Arc<candle_core::tensor::Tensor_>>
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
  18: core::ptr::drop_in_place<candle_core::tensor::Tensor>
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
  19: mistralrs_core::models::quantized_llama::LayerWeights::forward_attn
             at /root/.cargo/git/checkouts/mistral.rs-0a2607fe9768eac5/a702c6d/mistralrs-core/src/models/quantized_llama.rs:213:5
  20: mistralrs_core::models::quantized_llama::ModelWeights::forward
             at /root/.cargo/git/checkouts/mistral.rs-0a2607fe9768eac5/a702c6d/mistralrs-core/src/models/quantized_llama.rs:657:24
  21: <mistralrs_core::pipeline::gguf::GGUFPipeline as mistralrs_core::pipeline::Pipeline>::forward_inputs
             at /root/.cargo/git/checkouts/mistral.rs-0a2607fe9768eac5/a702c6d/mistralrs-core/src/pipeline/gguf.rs:664:40
  22: mistralrs_core::pipeline::Pipeline::step::{{closure}}
             at /root/.cargo/git/checkouts/mistral.rs-0a2607fe9768eac5/a702c6d/mistralrs-core/src/pipeline/mod.rs:327:38
  23: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/future/future.rs:123:9
  24: mistralrs_core::engine::Engine::run::{{closure}}
             at /root/.cargo/git/checkouts/mistral.rs-0a2607fe9768eac5/a702c6d/mistralrs-core/src/engine/mod.rs:234:34
  25: mistralrs_core::MistralRs::new::{{closure}}::{{closure}}
             at /root/.cargo/git/checkouts/mistral.rs-0a2607fe9768eac5/a702c6d/mistralrs-core/src/lib.rs:332:30
  26: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/future/future.rs:123:9
  27: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
  28: tokio::runtime::coop::with_budget
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
  29: tokio::runtime::coop::budget
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  30: tokio::runtime::park::CachedParkThread::block_on
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
  31: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  32: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
  33: tokio::runtime::context::runtime::enter_runtime
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  34: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
  35: tokio::runtime::runtime::Runtime::block_on_inner
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs:363:45
  36: tokio::runtime::runtime::Runtime::block_on
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs:333:13
  37: mistralrs_core::MistralRs::new::{{closure}}
             at /root/.cargo/git/checkouts/mistral.rs-0a2607fe9768eac5/a702c6d/mistralrs-core/src/lib.rs:320:13
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread '<unnamed>' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cudarc-0.12.1/src/driver/safe/core.rs:252:76:
called `Result::unwrap()` on an `Err` value: DriverError(CUDA_ERROR_ILLEGAL_ADDRESS, "an illegal memory access was encountered")
stack backtrace:
   0:     0x61ce90cceff5 - std::backtrace_rs::backtrace::libunwind::trace::hc79cced6f418596d
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/../../backtrace/src/backtrace/libunwind.rs:105:5
   1:     0x61ce90cceff5 - std::backtrace_rs::backtrace::trace_unsynchronized::h06f3eef6c8a22cf0
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x61ce90cceff5 - std::sys_common::backtrace::_print_fmt::hba273d0c77fc3421
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/sys_common/backtrace.rs:68:5
   3:     0x61ce90cceff5 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h409f1e3c1e32650e
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x61ce90cfdecb - core::fmt::rt::Argument::fmt::h8811fe3c91cda7b3
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/fmt/rt.rs:142:9
   5:     0x61ce90cfdecb - core::fmt::write::h7a8f70a9b146d9ee
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/fmt/mod.rs:1153:17
   6:     0x61ce90ccaebf - std::io::Write::write_fmt::hc57d86a7c88c29ef
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/io/mod.rs:1843:15
   7:     0x61ce90ccedce - std::sys_common::backtrace::_print::h0dc0bbf9b429a58b
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x61ce90ccedce - std::sys_common::backtrace::print::hf60182bd4aee207d
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x61ce90cd08d9 - std::panicking::default_hook::{{closure}}::hd90db44a41f772dc
  10:     0x61ce90cd0579 - std::panicking::default_hook::hd86be16b87521210
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/panicking.rs:288:9
  11:     0x61ce8ddbd74a - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h0f4e2b1213798605
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/alloc/src/boxed.rs:2032:9
  12:     0x61ce8ddbd74a - test::test_main::{{closure}}::hec81cefc5baa15e2
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/test/src/lib.rs:138:21
  13:     0x61ce90cd0eac - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h2fe2a6e53d9884ad
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/alloc/src/boxed.rs:2032:9
  14:     0x61ce90cd0eac - std::panicking::rust_panic_with_hook::ha4f8caa112a16574
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/panicking.rs:792:13
  15:     0x61ce90cd0c56 - std::panicking::begin_panic_handler::{{closure}}::hc879855deab44ed0
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/panicking.rs:657:13
  16:     0x61ce90ccf4b9 - std::sys_common::backtrace::__rust_end_short_backtrace::h85e59f289fdfff6c
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/sys_common/backtrace.rs:171:18
  17:     0x61ce90cd0987 - rust_begin_unwind
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/panicking.rs:645:5
  18:     0x61ce8dc6f766 - core::panicking::panic_fmt::h0baef2c59e253f8d
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/panicking.rs:72:14
  19:     0x61ce8dc6fcf6 - core::result::unwrap_failed::ha3431373f2eea71f
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/result.rs:1654:5
  20:     0x61ce8edbe3fa - core::result::Result<T,E>::unwrap::h0fec05548d92e9c5
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/result.rs:1077:23
  21:     0x61ce8edbe3fa - <cudarc::driver::safe::core::CudaSlice<T> as core::ops::drop::Drop>::drop::he17e9948e4e2d725
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cudarc-0.12.1/src/driver/safe/core.rs:252:17
  22:     0x61ce8ebf6fe7 - core::ptr::drop_in_place<cudarc::driver::safe::core::CudaSlice<f32>>::h266239d2bab626e3
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
  23:     0x61ce8ebf63de - core::ptr::drop_in_place<candle_core::cuda_backend::CudaStorageSlice>::hca12f5525f1761be
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
  24:     0x61ce8ebf5ad7 - core::ptr::drop_in_place<candle_core::cuda_backend::CudaStorage>::hc8a04298e2362e61
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
  25:     0x61ce8ebf406c - core::ptr::drop_in_place<candle_core::storage::Storage>::h9a86953112c3957c
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
  26:     0x61ce8ebf855b - core::ptr::drop_in_place<core::cell::UnsafeCell<candle_core::storage::Storage>>::hf62c89b6f9218f5e
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
  27:     0x61ce8ebf8e7f - core::ptr::drop_in_place<std::sync::rwlock::RwLock<candle_core::storage::Storage>>::h390c39dc983e637a
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
  28:     0x61ce8ec7d85f - alloc::sync::Arc<T,A>::drop_slow::h8d79124923f6a52a
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/alloc/src/sync.rs:1804:18
  29:     0x61ce8ec9b302 - <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop::h4554c7a45f9a01c6
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/alloc/src/sync.rs:2462:13
  30:     0x61ce8ebec02b - core::ptr::drop_in_place<alloc::sync::Arc<std::sync::rwlock::RwLock<candle_core::storage::Storage>>>::h4b9b02fc6af06a2b
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
  31:     0x61ce8ebf3e5e - core::ptr::drop_in_place<candle_core::tensor::Tensor_>::h969af278012e4074
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
  32:     0x61ce8ec7d8ff - alloc::sync::Arc<T,A>::drop_slow::hc6742b7c6f00a1f3
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/alloc/src/sync.rs:1804:18
  33:     0x61ce8ec9b282 - <alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop::h225f1d395d75af8e
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/alloc/src/sync.rs:2462:13
  34:     0x61ce8ebf7b7b - core::ptr::drop_in_place<alloc::sync::Arc<candle_core::tensor::Tensor_>>::h35190ac780d405bd
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
  35:     0x61ce8ebf37eb - core::ptr::drop_in_place<candle_core::tensor::Tensor>::h9db9f54123674e57
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ptr/mod.rs:514:1
  36:     0x61ce8dff8ceb - mistralrs_core::models::quantized_llama::LayerWeights::forward_attn::h2c8f2cb2120c3544
                               at /root/.cargo/git/checkouts/mistral.rs-0a2607fe9768eac5/a702c6d/mistralrs-core/src/models/quantized_llama.rs:213:5
  37:     0x61ce8e007a22 - mistralrs_core::models::quantized_llama::ModelWeights::forward::h926345e9b57a5875
                               at /root/.cargo/git/checkouts/mistral.rs-0a2607fe9768eac5/a702c6d/mistralrs-core/src/models/quantized_llama.rs:657:24
  38:     0x61ce8e2329b5 - <mistralrs_core::pipeline::gguf::GGUFPipeline as mistralrs_core::pipeline::Pipeline>::forward_inputs::h243c075b9689b2e3
                               at /root/.cargo/git/checkouts/mistral.rs-0a2607fe9768eac5/a702c6d/mistralrs-core/src/pipeline/gguf.rs:664:40
  39:     0x61ce8e0d25e0 - mistralrs_core::pipeline::Pipeline::step::{{closure}}::hd84f65b1125e8bcf
                               at /root/.cargo/git/checkouts/mistral.rs-0a2607fe9768eac5/a702c6d/mistralrs-core/src/pipeline/mod.rs:327:38
  40:     0x61ce8df6fc04 - <core::pin::Pin<P> as core::future::future::Future>::poll::h9130a87a00b49342
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/future/future.rs:123:9
  41:     0x61ce8e0fb837 - mistralrs_core::engine::Engine::run::{{closure}}::hfe4716020979fa45
                               at /root/.cargo/git/checkouts/mistral.rs-0a2607fe9768eac5/a702c6d/mistralrs-core/src/engine/mod.rs:234:34
  42:     0x61ce8e261989 - mistralrs_core::MistralRs::new::{{closure}}::{{closure}}::h00d8b6d8f22531e1
                               at /root/.cargo/git/checkouts/mistral.rs-0a2607fe9768eac5/a702c6d/mistralrs-core/src/lib.rs:332:30
  43:     0x61ce8df6fa97 - <core::pin::Pin<P> as core::future::future::Future>::poll::h5d182315eb8fc5ad
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/future/future.rs:123:9
  44:     0x61ce8e191146 - tokio::runtime::park::CachedParkThread::block_on::{{closure}}::h41751e0de0933807
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
  45:     0x61ce8e190a7b - tokio::runtime::coop::with_budget::h48a841bb411d59bc
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
  46:     0x61ce8e190a7b - tokio::runtime::coop::budget::hc5aa5ffdea92f45b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  47:     0x61ce8e190a7b - tokio::runtime::park::CachedParkThread::block_on::hc7505029a4e7c65f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
  48:     0x61ce8e02c534 - tokio::runtime::context::blocking::BlockingRegionGuard::block_on::hc8996903afa00fb9
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  49:     0x61ce8e14c18f - tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}::hed9b49b2adfd1ed5
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
  50:     0x61ce8dfa1dd3 - tokio::runtime::context::runtime::enter_runtime::h8241746f6a1640be
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  51:     0x61ce8e14c01a - tokio::runtime::scheduler::multi_thread::MultiThread::block_on::hd77c8424eaba7afe
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
  52:     0x61ce8e0e030a - tokio::runtime::runtime::Runtime::block_on_inner::h60341cd647cf4b48
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs:363:45
  53:     0x61ce8e0e091b - tokio::runtime::runtime::Runtime::block_on::he30078dafd5026ae
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs:333:13
  54:     0x61ce8e26167d - mistralrs_core::MistralRs::new::{{closure}}::hed7f0cfc4955b5ad
                               at /root/.cargo/git/checkouts/mistral.rs-0a2607fe9768eac5/a702c6d/mistralrs-core/src/lib.rs:320:13
  55:     0x61ce8e1f6fb6 - std::sys_common::backtrace::__rust_begin_short_backtrace::h2587321759660118
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/sys_common/backtrace.rs:155:18
  56:     0x61ce8df62eb1 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hceae8b5bc8aedaff
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/thread/mod.rs:523:17
  57:     0x61ce8e1cce41 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h70f0fff85dfd44ed
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/panic/unwind_safe.rs:272:9
  58:     0x61ce8e185ad1 - std::panicking::try::do_call::h671fb002c7e08351
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/panicking.rs:552:40
  59:     0x61ce8e1cc3db - __rust_try
  60:     0x61ce8e184f02 - std::panicking::try::h4a59a8198e7f3a4d
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/panicking.rs:516:19
  61:     0x61ce8df627d5 - std::panic::catch_unwind::hbe987f130fbd4512
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/panic.rs:149:14
  62:     0x61ce8df627d5 - std::thread::Builder::spawn_unchecked_::{{closure}}::h90ae6a7629d754ea
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/thread/mod.rs:522:30
  63:     0x61ce8df05a4f - core::ops::function::FnOnce::call_once{{vtable.shim}}::h3f854c64849c60cc
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/core/src/ops/function.rs:250:5
  64:     0x61ce90cd624b - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h5cf039e566d31df2
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/alloc/src/boxed.rs:2018:9
  65:     0x61ce90cd624b - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h5b8a7e7667fbf80b
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/alloc/src/boxed.rs:2018:9
  66:     0x61ce90cd624b - std::sys::pal::unix::thread::Thread::new::thread_start::h47ad6cb551091e6a
                               at /rustc/9d5cdf75aa42faaf0b58ba21a510117e8d0051a3/library/std/src/sys/pal/unix/thread.rs:108:17
  67:     0x7d57f3c70ac3 - <unknown>
  68:     0x7d57f3d01a04 - __clone
  69:                0x0 - <unknown>
thread '<unnamed>' panicked at library/core/src/panicking.rs:223:5:
panic in a destructor during cleanup
thread caused non-unwinding panic. aborting.

Tried the latest commit from today and using Llama3_1_8bInstruct.

mistralrs={git="https://github.com/EricLBuehler/mistral.rs.git", features=["cuda", "cudnn"], optional=true, rev="a702c6dd2944aaf75800b11f4dfeec6fe5a9b068"}

Originally posted by @ShelbyJenkins in #651 (comment)

[EricLBuehler]

@ShelbyJenkins can you reproduce the issue if you ensure running without PagedAttention?

[ShelbyJenkins]

Sorry for the delay. Been updating things on my backend. I just upgraded to the newest hash. Love the new API <3

Yes, PagedAttention should be disabled based on how I init it right? Additionally, I'm feeding it a mapper, so it should disable it by default.

let pipeline: std::sync::Arc<tokio::sync::Mutex<dyn Pipeline + Send + Sync>> = loader.load_model_from_path(
            &paths,
            &ModelDType::Auto,
            &device,
            false,
            mapper,
            None,
            None,
        )?;
        
MistralRsBuilder::new(
            pipeline,
            SchedulerConfig::DefaultScheduler {
                method: DefaultSchedulerMethod::Fixed(5.try_into().unwrap()),
            },
        )
        
        ```

[ShelbyJenkins]

Interestingly, phi3.5 works with my setup.

Mistral Nemo and Llama3.2 3b do have the CUDA_ERROR however.

[EricLBuehler]

@ShelbyJenkins I'll take a look at what is causing this.