Llama 3.2 on macOS: "Metal contiguous affine I64 not implemented"

[ChristianWeyer]

Describe the bug

Running Llama 3.2 on my MacBook Pro M3 Max (128GB) - with

cargo run --release --features metal -- --port 1234 vision-plain -m lamm-mit/Cephalo-L
lama-3.2-11B-Vision-Instruct-128k -a vllama

Posting an image with

from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:1234/v1/"
)

completion = client.chat.completions.create(
    model="llama-vision",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://www.nhmagazine.com/content/uploads/2019/05/mtwashingtonFranconia-2-19-18-108-Edit-Edit.jpg"
                    },
                },
                {
                    "type": "text",
                    "text": "What is shown in this image? Write a detailed response analyzing the scene.",
                },
            ],
        }
    ],
    max_tokens=256,
    frequency_penalty=1.0,
    top_p=0.1,
    temperature=0
)

resp = completion.choices[0].message.content
print(resp)

Gives this error:
ERROR mistralrs_core::engine: prompt step - Model failed with error: Msg("Metal contiguous affine I64 not implemented")

Latest commit or version

ce02618

@EricLBuehler

[EricLBuehler]

@ChristianWeyer, can you please run without --release and let me know what the output is when run with RUST_BACKTRACE=full?

[ChristianWeyer]

Here we go @EricLBuehler :

Error...

2024-09-30T15:13:02.669854Z ERROR mistralrs_core::engine: prompt step - Model failed with error: WithBacktrace { inner: Msg("Metal contiguous affine I64 not implemented"), backtrace: Backtrace [{ fn: "std::backtrace_rs::backtrace::libunwind::trace", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs", line: 116 }, { fn: "std::backtrace_rs::backtrace::trace_unsynchronized", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs", line: 66 }, { fn: "std::backtrace::Backtrace::create", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/backtrace.rs", line: 331 }, { fn: "candle_core::error::Error::bt", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/error.rs", line: 266 }, { fn: "::affine", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/error.rs", line: 288 }, { fn: "candle_core::storage::Storage::affine", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/storage.rs", line: 87 }, { fn: "candle_core::tensor::Tensor::affine", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/tensor.rs", line: 781 }, { fn: "candle_core::tensor:: for f64>::sub", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/tensor.rs", line: 2630 }, { fn: "mistralrs_core::vision_models::mllama::vision::_prepare_aspect_ratio_attention_mask", file: "./mistralrs-core/src/vision_models/mllama/vision.rs", line: 356 }, { fn: "mistralrs_core::vision_models::mllama::vision::MLlamaVisionModel::forward", file: "./mistralrs-core/src/vision_models/mllama/vision.rs", line: 524 }, { fn: "mistralrs_core::vision_models::mllama::MLlamaModel::forward_inner", file: "./mistralrs-core/src/vision_models/mllama/mod.rs", line: 137 }, { fn: "::forward", file: "./mistralrs-core/src/vision_models/mllama/mod.rs", line: 213 }, { fn: "::forward_inputs", file: "./mistralrs-core/src/pipeline/vision.rs", line: 458 }, { fn: "mistralrs_core::pipeline::Pipeline::step::{{closure}}", file: "./mistralrs-core/src/pipeline/mod.rs", line: 344 }, { fn: " as core::future::future::Future>::poll", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/future/future.rs", line: 123 }, { fn: "mistralrs_core::engine::Engine::run::{{closure}}", file: "./mistralrs-core/src/engine/mod.rs", line: 241 }, { fn: "mistralrs_core::MistralRs::new::{{closure}}::{{closure}}", file: "./mistralrs-core/src/lib.rs", line: 339 }, { fn: " as core::future::future::Future>::poll", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/future/future.rs", line: 123 }, { fn: "tokio::runtime::park::CachedParkThread::block_on::{{closure}}", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs", line: 281 }, { fn: "tokio::runtime::coop::with_budget", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs", line: 107 }, { fn: "tokio::runtime::coop::budget", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs", line: 73 }, { fn: "tokio::runtime::park::CachedParkThread::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs", line: 281 }, { fn: "tokio::runtime::context::blocking::BlockingRegionGuard::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs", line: 66 }, { fn: "tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs", line: 87 }, { fn: "tokio::runtime::context::runtime::enter_runtime", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs", line: 65 }, { fn: "tokio::runtime::scheduler::multi_thread::MultiThread::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs", line: 86 }, { fn: "tokio::runtime::runtime::Runtime::block_on_inner", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs", line: 363 }, { fn: "tokio::runtime::runtime::Runtime::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs", line: 333 }, { fn: "mistralrs_core::MistralRs::new::{{closure}}", file: "./mistralrs-core/src/lib.rs", line: 327 }, { fn: "std::sys::backtrace::__rust_begin_short_backtrace", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs", line: 152 }, { fn: "std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/mod.rs", line: 538 }, { fn: " as core::ops::function::FnOnce<()>>::call_once", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs", line: 272 }, { fn: "std::panicking::try::do_call", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs", line: 557 }, { fn: "___rust_try" }, { fn: "std::panicking::try", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs", line: 521 }, { fn: "std::panic::catch_unwind", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs", line: 350 }, { fn: "std::thread::Builder::spawn_unchecked_::{{closure}}", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/mod.rs", line: 537 }, { fn: "core::ops::function::FnOnce::call_once{{vtable.shim}}", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/ops/function.rs", line: 250 }, { fn: " as core::ops::function::FnOnce>::call_once", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs", line: 2070 }, { fn: " as core::ops::function::FnOnce>::call_once", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs", line: 2070 }, { fn: "std::sys::pal::unix::thread::Thread::new::thread_start", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/pal/unix/thread.rs", line: 108 }, { fn: "__pthread_deallocate" }] }
2024-09-30T15:13:07.330114Z ERROR mistralrs_core::engine: prompt step - Model failed with error: WithBacktrace { inner: Msg("Metal contiguous affine I64 not implemented"), backtrace: Backtrace [{ fn: "std::backtrace_rs::backtrace::libunwind::trace", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs", line: 116 }, { fn: "std::backtrace_rs::backtrace::trace_unsynchronized", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs", line: 66 }, { fn: "std::backtrace::Backtrace::create", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/backtrace.rs", line: 331 }, { fn: "candle_core::error::Error::bt", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/error.rs", line: 266 }, { fn: "::affine", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/error.rs", line: 288 }, { fn: "candle_core::storage::Storage::affine", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/storage.rs", line: 87 }, { fn: "candle_core::tensor::Tensor::affine", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/tensor.rs", line: 781 }, { fn: "candle_core::tensor:: for f64>::sub", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/tensor.rs", line: 2630 }, { fn: "mistralrs_core::vision_models::mllama::vision::_prepare_aspect_ratio_attention_mask", file: "./mistralrs-core/src/vision_models/mllama/vision.rs", line: 356 }, { fn: "mistralrs_core::vision_models::mllama::vision::MLlamaVisionModel::forward", file: "./mistralrs-core/src/vision_models/mllama/vision.rs", line: 524 }, { fn: "mistralrs_core::vision_models::mllama::MLlamaModel::forward_inner", file: "./mistralrs-core/src/vision_models/mllama/mod.rs", line: 137 }, { fn: "::forward", file: "./mistralrs-core/src/vision_models/mllama/mod.rs", line: 213 }, { fn: "::forward_inputs", file: "./mistralrs-core/src/pipeline/vision.rs", line: 458 }, { fn: "mistralrs_core::pipeline::Pipeline::step::{{closure}}", file: "./mistralrs-core/src/pipeline/mod.rs", line: 344 }, { fn: " as core::future::future::Future>::poll", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/future/future.rs", line: 123 }, { fn: "mistralrs_core::engine::Engine::run::{{closure}}", file: "./mistralrs-core/src/engine/mod.rs", line: 241 }, { fn: "mistralrs_core::MistralRs::new::{{closure}}::{{closure}}", file: "./mistralrs-core/src/lib.rs", line: 339 }, { fn: " as core::future::future::Future>::poll", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/future/future.rs", line: 123 }, { fn: "tokio::runtime::park::CachedParkThread::block_on::{{closure}}", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs", line: 281 }, { fn: "tokio::runtime::coop::with_budget", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs", line: 107 }, { fn: "tokio::runtime::coop::budget", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs", line: 73 }, { fn: "tokio::runtime::park::CachedParkThread::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs", line: 281 }, { fn: "tokio::runtime::context::blocking::BlockingRegionGuard::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs", line: 66 }, { fn: "tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs", line: 87 }, { fn: "tokio::runtime::context::runtime::enter_runtime", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs", line: 65 }, { fn: "tokio::runtime::scheduler::multi_thread::MultiThread::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs", line: 86 }, { fn: "tokio::runtime::runtime::Runtime::block_on_inner", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs", line: 363 }, { fn: "tokio::runtime::runtime::Runtime::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs", line: 333 }, { fn: "mistralrs_core::MistralRs::new::{{closure}}", file: "./mistralrs-core/src/lib.rs", line: 327 }, { fn: "std::sys::backtrace::__rust_begin_short_backtrace", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs", line: 152 }, { fn: "std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/mod.rs", line: 538 }, { fn: " as core::ops::function::FnOnce<()>>::call_once", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs", line: 272 }, { fn: "std::panicking::try::do_call", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs", line: 557 }, { fn: "___rust_try" }, { fn: "std::panicking::try", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs", line: 521 }, { fn: "std::panic::catch_unwind", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs", line: 350 }, { fn: "std::thread::Builder::spawn_unchecked_::{{closure}}", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/mod.rs", line: 537 }, { fn: "core::ops::function::FnOnce::call_once{{vtable.shim}}", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/ops/function.rs", line: 250 }, { fn: " as core::ops::function::FnOnce>::call_once", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs", line: 2070 }, { fn: " as core::ops::function::FnOnce>::call_once", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs", line: 2070 }, { fn: "std::sys::pal::unix::thread::Thread::new::thread_start", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/pal/unix/thread.rs", line: 108 }, { fn: "__pthread_deallocate" }] }
2024-09-30T15:13:12.222865Z ERROR mistralrs_core::engine: prompt step - Model failed with error: WithBacktrace { inner: Msg("Metal contiguous affine I64 not implemented"), backtrace: Backtrace [{ fn: "std::backtrace_rs::backtrace::libunwind::trace", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs", line: 116 }, { fn: "std::backtrace_rs::backtrace::trace_unsynchronized", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs", line: 66 }, { fn: "std::backtrace::Backtrace::create", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/backtrace.rs", line: 331 }, { fn: "candle_core::error::Error::bt", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/error.rs", line: 266 }, { fn: "::affine", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/error.rs", line: 288 }, { fn: "candle_core::storage::Storage::affine", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/storage.rs", line: 87 }, { fn: "candle_core::tensor::Tensor::affine", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/tensor.rs", line: 781 }, { fn: "candle_core::tensor:: for f64>::sub", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/tensor.rs", line: 2630 }, { fn: "mistralrs_core::vision_models::mllama::vision::_prepare_aspect_ratio_attention_mask", file: "./mistralrs-core/src/vision_models/mllama/vision.rs", line: 356 }, { fn: "mistralrs_core::vision_models::mllama::vision::MLlamaVisionModel::forward", file: "./mistralrs-core/src/vision_models/mllama/vision.rs", line: 524 }, { fn: "mistralrs_core::vision_models::mllama::MLlamaModel::forward_inner", file: "./mistralrs-core/src/vision_models/mllama/mod.rs", line: 137 }, { fn: "::forward", file: "./mistralrs-core/src/vision_models/mllama/mod.rs", line: 213 }, { fn: "::forward_inputs", file: "./mistralrs-core/src/pipeline/vision.rs", line: 458 }, { fn: "mistralrs_core::pipeline::Pipeline::step::{{closure}}", file: "./mistralrs-core/src/pipeline/mod.rs", line: 344 }, { fn: " as core::future::future::Future>::poll", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/future/future.rs", line: 123 }, { fn: "mistralrs_core::engine::Engine::run::{{closure}}", file: "./mistralrs-core/src/engine/mod.rs", line: 241 }, { fn: "mistralrs_core::MistralRs::new::{{closure}}::{{closure}}", file: "./mistralrs-core/src/lib.rs", line: 339 }, { fn: " as core::future::future::Future>::poll", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/future/future.rs", line: 123 }, { fn: "tokio::runtime::park::CachedParkThread::block_on::{{closure}}", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs", line: 281 }, { fn: "tokio::runtime::coop::with_budget", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs", line: 107 }, { fn: "tokio::runtime::coop::budget", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs", line: 73 }, { fn: "tokio::runtime::park::CachedParkThread::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs", line: 281 }, { fn: "tokio::runtime::context::blocking::BlockingRegionGuard::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs", line: 66 }, { fn: "tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs", line: 87 }, { fn: "tokio::runtime::context::runtime::enter_runtime", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs", line: 65 }, { fn: "tokio::runtime::scheduler::multi_thread::MultiThread::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs", line: 86 }, { fn: "tokio::runtime::runtime::Runtime::block_on_inner", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs", line: 363 }, { fn: "tokio::runtime::runtime::Runtime::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs", line: 333 }, { fn: "mistralrs_core::MistralRs::new::{{closure}}", file: "./mistralrs-core/src/lib.rs", line: 327 }, { fn: "std::sys::backtrace::__rust_begin_short_backtrace", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs", line: 152 }, { fn: "std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/mod.rs", line: 538 }, { fn: " as core::ops::function::FnOnce<()>>::call_once", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs", line: 272 }, { fn: "std::panicking::try::do_call", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs", line: 557 }, { fn: "___rust_try" }, { fn: "std::panicking::try", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs", line: 521 }, { fn: "std::panic::catch_unwind", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs", line: 350 }, { fn: "std::thread::Builder::spawn_unchecked_::{{closure}}", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/mod.rs", line: 537 }, { fn: "core::ops::function::FnOnce::call_once{{vtable.shim}}", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/ops/function.rs", line: 250 }, { fn: " as core::ops::function::FnOnce>::call_once", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs", line: 2070 }, { fn: " as core::ops::function::FnOnce>::call_once", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs", line: 2070 }, { fn: "std::sys::pal::unix::thread::Thread::new::thread_start", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/pal/unix/thread.rs", line: 108 }, { fn: "__pthread_deallocate" }] }

[EricLBuehler]

@ChristianWeyer thanks for the info! The error comes from this line:

mistral.rs/mistralrs-core/src/vision_models/mllama/vision.rs

Line 356 in ce02618

attention_mask = (1. - attention_mask)?;

I'll add a fix!

[EricLBuehler]

@ChristianWeyer I just merged #811 which should fix this, can you please confirm?

[ChristianWeyer]

New error @EricLBuehler:

2024-10-01T06:46:14.458166Z ERROR mistralrs_core::engine: prompt step - Model failed with error: WithBacktrace { inner: Msg("Metal contiguous index_select U32 I64 not implemented"), backtrace: Backtrace [{ fn: "std::backtrace_rs::backtrace::libunwind::trace", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs", line: 116 }, { fn: "std::backtrace_rs::backtrace::trace_unsynchronized", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs", line: 66 }, { fn: "std::backtrace::Backtrace::create", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/backtrace.rs", line: 331 }, { fn: "candle_core::error::Error::bt", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/error.rs", line: 266 }, { fn: "<candle_core::metal_backend::MetalStorage as candle_core::backend::BackendStorage>::index_select", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/error.rs", line: 288 }, { fn: "candle_core::storage::Storage::index_select", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/storage.rs", line: 696 }, { fn: "candle_core::tensor_indexing::<impl candle_core::tensor::Tensor>::index_select", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/tensor_indexing.rs", line: 368 }, { fn: "mistralrs_core::vision_models::mllama::repeat_interleave", file: "./mistralrs-core/src/vision_models/mllama/mod.rs", line: 39 }, { fn: "mistralrs_core::vision_models::mllama::prepare_cross_attention_mask", file: "./mistralrs-core/src/vision_models/mllama/mod.rs", line: 50 }, { fn: "mistralrs_core::vision_models::mllama::MLlamaModel::forward_inner", file: "./mistralrs-core/src/vision_models/mllama/mod.rs", line: 148 }, { fn: "<mistralrs_core::vision_models::mllama::MLlamaModel as mistralrs_core::pipeline::loaders::vision_loaders::VisionModel>::forward", file: "./mistralrs-core/src/vision_models/mllama/mod.rs", line: 211 }, { fn: "<mistralrs_core::pipeline::vision::VisionPipeline as mistralrs_core::pipeline::Pipeline>::forward_inputs", file: "./mistralrs-core/src/pipeline/vision.rs", line: 458 }, { fn: "mistralrs_core::pipeline::Pipeline::step::{{closure}}", file: "./mistralrs-core/src/pipeline/mod.rs", line: 344 }, { fn: "<core::pin::Pin<P> as core::future::future::Future>::poll", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/future/future.rs", line: 123 }, { fn: "mistralrs_core::engine::Engine::run::{{closure}}", file: "./mistralrs-core/src/engine/mod.rs", line: 241 }, { fn: "mistralrs_core::MistralRs::new::{{closure}}::{{closure}}", file: "./mistralrs-core/src/lib.rs", line: 339 }, { fn: "<core::pin::Pin<P> as core::future::future::Future>::poll", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/future/future.rs", line: 123 }, { fn: "tokio::runtime::park::CachedParkThread::block_on::{{closure}}", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs", line: 281 }, { fn: "tokio::runtime::coop::with_budget", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs", line: 107 }, { fn: "tokio::runtime::coop::budget", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs", line: 73 }, { fn: "tokio::runtime::park::CachedParkThread::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs", line: 281 }, { fn: "tokio::runtime::context::blocking::BlockingRegionGuard::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs", line: 66 }, { fn: "tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs", line: 87 }, { fn: "tokio::runtime::context::runtime::enter_runtime", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs", line: 65 }, { fn: "tokio::runtime::scheduler::multi_thread::MultiThread::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs", line: 86 }, { fn: "tokio::runtime::runtime::Runtime::block_on_inner", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs", line: 363 }, { fn: "tokio::runtime::runtime::Runtime::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs", line: 333 }, { fn: "mistralrs_core::MistralRs::new::{{closure}}", file: "./mistralrs-core/src/lib.rs", line: 327 }, { fn: "std::sys::backtrace::__rust_begin_short_backtrace", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs", line: 152 }, { fn: "std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/mod.rs", line: 538 }, { fn: "<core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs", line: 272 }, { fn: "std::panicking::try::do_call", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs", line: 557 }, { fn: "___rust_try" }, { fn: "std::panicking::try", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs", line: 521 }, { fn: "std::panic::catch_unwind", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs", line: 350 }, { fn: "std::thread::Builder::spawn_unchecked_::{{closure}}", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/mod.rs", line: 537 }, { fn: "core::ops::function::FnOnce::call_once{{vtable.shim}}", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/ops/function.rs", line: 250 }, { fn: "<alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs", line: 2070 }, { fn: "<alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs", line: 2070 }, { fn: "std::sys::pal::unix::thread::Thread::new::thread_start", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/pal/unix/thread.rs", line: 108 }, { fn: "__pthread_deallocate" }] }

[EricLBuehler]

@ChristianWeyer I just merged #812 which should fix that issue, can you please confirm if it runs now?

[ChristianWeyer]

Nope @EricLBuehler:

2024-10-01T17:31:06.405826Z ERROR mistralrs_core::engine: prompt step - Model failed with error: WithBacktrace { inner: Msg("Metal strided to_dtype F64 F16 not implemented"), backtrace: Backtrace [{ fn: "std::backtrace_rs::backtrace::libunwind::trace", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs", line: 116 }, { fn: "std::backtrace_rs::backtrace::trace_unsynchronized", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs", line: 66 }, { fn: "std::backtrace::Backtrace::create", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/backtrace.rs", line: 331 }, { fn: "candle_core::error::Error::bt", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/error.rs", line: 266 }, { fn: "<candle_core::metal_backend::MetalStorage as candle_core::backend::BackendStorage>::to_dtype", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/error.rs", line: 288 }, { fn: "candle_core::storage::Storage::to_dtype", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/storage.rs", line: 190 }, { fn: "candle_core::tensor::Tensor::to_dtype", file: "/Users/christianweyer/.cargo/git/checkouts/candle-c6a149c3b35a488f/9c62368/candle-core/src/tensor.rs", line: 2084 }, { fn: "mistralrs_core::layers_masker::masked_fill", file: "./mistralrs-core/src/layers_masker.rs", line: 27 }, { fn: "mistralrs_core::vision_models::mllama::prepare_cross_attention_mask", file: "./mistralrs-core/src/vision_models/mllama/mod.rs", line: 63 }, { fn: "mistralrs_core::vision_models::mllama::MLlamaModel::forward_inner", file: "./mistralrs-core/src/vision_models/mllama/mod.rs", line: 154 }, { fn: "<mistralrs_core::vision_models::mllama::MLlamaModel as mistralrs_core::pipeline::loaders::vision_loaders::VisionModel>::forward", file: "./mistralrs-core/src/vision_models/mllama/mod.rs", line: 217 }, { fn: "<mistralrs_core::pipeline::vision::VisionPipeline as mistralrs_core::pipeline::Pipeline>::forward_inputs", file: "./mistralrs-core/src/pipeline/vision.rs", line: 458 }, { fn: "mistralrs_core::pipeline::Pipeline::step::{{closure}}", file: "./mistralrs-core/src/pipeline/mod.rs", line: 344 }, { fn: "<core::pin::Pin<P> as core::future::future::Future>::poll", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/future/future.rs", line: 123 }, { fn: "mistralrs_core::engine::Engine::run::{{closure}}", file: "./mistralrs-core/src/engine/mod.rs", line: 241 }, { fn: "mistralrs_core::MistralRs::new::{{closure}}::{{closure}}", file: "./mistralrs-core/src/lib.rs", line: 339 }, { fn: "<core::pin::Pin<P> as core::future::future::Future>::poll", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/future/future.rs", line: 123 }, { fn: "tokio::runtime::park::CachedParkThread::block_on::{{closure}}", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs", line: 281 }, { fn: "tokio::runtime::coop::with_budget", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs", line: 107 }, { fn: "tokio::runtime::coop::budget", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs", line: 73 }, { fn: "tokio::runtime::park::CachedParkThread::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs", line: 281 }, { fn: "tokio::runtime::context::blocking::BlockingRegionGuard::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs", line: 66 }, { fn: "tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs", line: 87 }, { fn: "tokio::runtime::context::runtime::enter_runtime", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs", line: 65 }, { fn: "tokio::runtime::scheduler::multi_thread::MultiThread::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs", line: 86 }, { fn: "tokio::runtime::runtime::Runtime::block_on_inner", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs", line: 363 }, { fn: "tokio::runtime::runtime::Runtime::block_on", file: "/Users/christianweyer/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs", line: 333 }, { fn: "mistralrs_core::MistralRs::new::{{closure}}", file: "./mistralrs-core/src/lib.rs", line: 327 }, { fn: "std::sys::backtrace::__rust_begin_short_backtrace", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs", line: 152 }, { fn: "std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/mod.rs", line: 538 }, { fn: "<core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panic/unwind_safe.rs", line: 272 }, { fn: "std::panicking::try::do_call", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs", line: 557 }, { fn: "___rust_try" }, { fn: "std::panicking::try", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs", line: 521 }, { fn: "std::panic::catch_unwind", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panic.rs", line: 350 }, { fn: "std::thread::Builder::spawn_unchecked_::{{closure}}", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/mod.rs", line: 537 }, { fn: "core::ops::function::FnOnce::call_once{{vtable.shim}}", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/ops/function.rs", line: 250 }, { fn: "<alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs", line: 2070 }, { fn: "<alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs", line: 2070 }, { fn: "std::sys::pal::unix::thread::Thread::new::thread_start", file: "/rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/pal/unix/thread.rs", line: 108 }, { fn: "__pthread_deallocate" }] }

[EricLBuehler]

@ChristianWeyer #814 should fix that one :)

[ChristianWeyer]

Now, I do not see any logs in the terminal for the request.

The call takes 42 seconds and returns:
@EricLBuehler

{
  "id": "1",
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "message": {
        "content": "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!",
        "role": "assistant",
        "tool_calls": []
      },
      "logprobs": null
    }
  ],
  "created": 1727852481,
  "model": "lamm-mit/Cephalo-Llama-3.2-11B-Vision-Instruct-128k",
  "system_fingerprint": "local",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 257,
    "prompt_tokens": 27,
    "total_tokens": 284,
    "avg_tok_per_sec": 7.0387626,
    "avg_prompt_tok_per_sec": 1.3529766,
    "avg_compl_tok_per_sec": 12.602982,
    "total_time_sec": 40.348,
    "total_prompt_time_sec": 19.956,
    "total_completion_time_sec": 20.392
  }
}