DTypeMismatchBinaryOp on add w/ mixtral

[wseaton]

Describe the bug

Loading Mixtral-8x22b on 8xA100s via ISQ at Q8_0, model loads, and then at first token we get:

2024-08-30T16:00:20.769043Z ERROR mistralrs_core::engine: prompt step - Model failed with error: WithBacktrace { inner: DTypeMismatchBinaryOp { lhs: BF16, rhs: F32, op: "add" }, backtrace: Backtrace [{ fn: "candle_core::error::Error::bt" }, { fn: "candle_core::storage::Storage::same_dtype" }, { fn: "candle_core::tensor::Tensor::add" }, { fn: "mistralrs_core::models::mixtral::Model::forward" }, { fn: "<mistralrs_core::models::mixtral::Model as mistralrs_core::pipeline::loaders::normal_loaders::NormalModel>::forward" }, { fn: "<mistralrs_core::pipeline::normal::NormalPipeline as mistralrs_core::pipeline::Pipeline>::forward_inputs" }, { fn: "mistralrs_core::pipeline::Pipeline::step::{{closure}}" }, { fn: "mistralrs_core::engine::Engine::run::{{closure}}" }, { fn: "std::sys::backtrace::__rust_begin_short_backtrace" }, { fn: "core::ops::function::FnOnce::call_once{{vtable.shim}}" }, { fn: "std::sys::pal::unix::thread::Thread::new::thread_start" }, { fn: "start_thread" }, { fn: "...
thread '<unnamed>' panicked at mistralrs-core/src/engine/mod.rs:219:25:
called `Result::unwrap()` on an `Err` value: SendError { .. }
stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: core::result::unwrap_failed
3: mistralrs_core::engine::Engine::run::{{closure}}
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Latest commit or version

cfc0fa7

[wseaton]

Worth noting with HQQ8 I get a similar (but distinct) engine error:

2024-08-30T16:50:01.494400Z ERROR mistralrs_core::engine: prompt step - Model failed with error: WithBacktrace { inner: DTypeMismatchBinaryOp { lhs: BF16, rhs: F32, op: "mul" }, backtrace: Backtrace [{ fn: "candle_core::error::Error::bt" }, { fn: "candle_core::storage::Storage::same_dtype" }, { fn: "candle_core::tensor::Tensor::mul" }, { fn: "candle_core::tensor::Tensor::broadcast_mul" }, { fn: "mistralrs_core::models::mixtral::Model::forward" }, { fn: "<mistralrs_core::models::mixtral::Model as mistralrs_core::pipeline::loaders::normal_loaders::NormalModel>::forward" }, { fn: "<mistralrs_core::pipeline::normal::NormalPipeline as mistralrs_core::pipeline::Pipeline>::forward_inputs" }, { fn: "mistralrs_core::pipeline::Pipeline::step::{{closure}}" }, { fn: "mistralrs_core::engine::Engine::run::{{closure}}" }, { fn: "std::sys::backtrace::__rust_begin_short_backtrace" }, { fn: "core::ops::function::FnOnce::call_once{{vtable.shim}}" }, { fn: "std::sys::pal::unix::thread::Thread:...

[EricLBuehler]

Hi @wseaton! Thanks for reporting this. I'm not exactly sure where this is occurring, but I think there may still be some error here. I just added #730, can you please confirm if this fixes the issue?

[EricLBuehler]

@wseaton this is fixed now after I merged #730. Closing now, please feel free to reopen!

[wseaton]

ISQ is working now for mixtral, but I am running into an error similar to: #715. This happens upon first completion request.

[EricLBuehler]

@wseaton thanks for letting me know, I'll take a look at #715.