Fix Metal python package running on CPU

[EricLBuehler]

Refs #555. @KaQuMiQ I added some debug statements to get a better picture of what's going on. Can you please install from source:

(assuming you have Rust installed, which I think you do if you can run the PyPI package):

git clone https://github.com/EricLBuehler/mistral.rs.git
cd mistral.rs
git pull
git switch metal_on_cpu
< Activate a Python env now to install into >
pip install maturin[patchelf]
maturin develop -r --features metal

If you could try this out that would be great.

[github-actions]

Code Metrics Report

  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 C Header                2           35           28            0            7
 Dockerfile              1           34           25            0            9
 Happy                   1          442          369            0           73
 JSON                   11          102          101            0            1
 Python                 41         1586         1368           46          172
 TOML                   19          558          492           11           55
-------------------------------------------------------------------------------
 Jupyter Notebooks       2            0            0            0            0
 |- Markdown             2           77           32           31           14
 |- Python               2          196          169            1           26
 (Total)                            273          201           32           40
-------------------------------------------------------------------------------
 Markdown               24         1817            0         1371          446
 |- BASH                 5          101           98            0            3
 |- JSON                 1           12           12            0            0
 |- Python               5           92           82            0           10
 |- Rust                 6          407          364           19           24
 |- TOML                 2           75           63            0           12
 (Total)                           2504          619         1390          495
-------------------------------------------------------------------------------
 Rust                  169        54555        49536          970         4049
 |- Markdown            91          848           13          785           50
 (Total)                          55403        49549         1755         4099
===============================================================================
 Total                 271        59129        51919         2398         4812
===============================================================================
  

[KaQuMiQ]

I have failed to build the package that way. I have read your release script and modified manually Cargo.toml inside mistralrs-pyo3 folder and built there without installing patchelf (failed to install that one) - it seems to work now with metal acceleration. Here is build log:

> maturin develop -r --features metal
⚠️  Warning: You specified maturin ==1.4 in pyproject.toml under `build-system.requires`, but the current maturin version is 1.7.0
🔗 Found pyo3 bindings
🐍 Found CPython 3.12 at /Users/miquido/Projects/mistral.rs/.venv/bin/python
📡 Using build options profile from pyproject.toml
   Compiling pyo3-build-config v0.22.1
   Compiling serde_json v1.0.120
   Compiling tracing v0.1.40
   Compiling tower v0.4.13
   Compiling mistralrs-core v0.2.0 (/Users/miquido/Projects/mistral.rs/mistralrs-core)
   Compiling h2 v0.4.5
   Compiling candle-metal-kernels v0.6.0 (https://github.com/EricLBuehler/candle.git?rev=c967be9#c967be9c)
   Compiling tracing-subscriber v0.3.18
   Compiling pyo3-ffi v0.22.1
   Compiling pyo3-macros-backend v0.22.1
   Compiling pyo3 v0.22.1
   Compiling mistralrs-pyo3 v0.2.0 (/Users/miquido/Projects/mistral.rs/mistralrs-pyo3)
   Compiling safetensors v0.4.3
   Compiling ureq v2.10.0
   Compiling hyper v1.4.1
   Compiling askama v0.12.1
   Compiling candle-core v0.6.0 (https://github.com/EricLBuehler/candle.git?rev=c967be9#c967be9c)
   Compiling hyper-util v0.1.6
   Compiling plotly_kaleido v0.9.0
   Compiling pyo3-macros v0.22.1
   Compiling hyper-tls v0.6.0
   Compiling reqwest v0.12.5
   Compiling plotly v0.9.0
   Compiling hf-hub v0.3.2
   Compiling tokenizers v0.19.1
   Compiling candle-nn v0.6.0 (https://github.com/EricLBuehler/candle.git?rev=c967be9#c967be9c)
   Compiling mistralrs-vision v0.2.0 (/Users/miquido/Projects/mistral.rs/mistralrs-vision)
    Finished `release` profile [optimized] target(s) in 45.80s
📖 Found type stub file at mistralrs.pyi
📦 Built wheel for CPython 3.12 to /var/folders/9x/1vm9c8gj3jb_mm3b8yg2rrvw0000gn/T/.tmpeadSgs/mistralrs-0.2.0-cp312-cp312-macosx_11_0_arm64.whl
✏️  Setting installed package as editable
🛠 Installed mistralrs-0.2.0

Here is my modifierd Cargo.toml:

[package]
name = "mistralrs-pyo3"
authors = ["Eric Buehler"]
version.workspace = true
edition.workspace = true
description.workspace = true
repository.workspace = true
keywords.workspace = true
categories.workspace = true
license.workspace = true
homepage.workspace = true

[lib]
name = "mistralrs"
crate-type = ["cdylib"]
doc = false

[dependencies]
pyo3.workspace = true
mistralrs-core = { version = "0.2.0", path = "../mistralrs-core", features = ["pyo3_macros", "metal"] }
serde.workspace = true
serde_json.workspace = true
candle-core = { git = "https://github.com/EricLBuehler/candle.git", version = "0.6.0", rev = "c967be9", features=["metal"] }
indexmap.workspace = true
accelerate-src = { workspace = true, optional = true }
intel-mkl-src = { workspace = true, optional = true }
either.workspace = true
futures.workspace = true
tokio.workspace = true
image.workspace = true
reqwest.workspace = true
base64.workspace = true

[build-dependencies]
pyo3-build-config = "0.22"

[features]
cuda = ["candle-core/cuda", "mistralrs-core/cuda"]
cudnn = ["candle-core/cudnn", "mistralrs-core/cudnn"]
metal = ["candle-core/metal", "mistralrs-core/metal"]
flash-attn = ["cuda", "mistralrs-core/flash-attn"]
accelerate = ["mistralrs-core/accelerate"]
mkl = ["mistralrs-core/mkl"]

Updated two lines comparing to the original:

mistralrs-core = { version = "0.2.0", path = "../mistralrs-core", features = ["pyo3_macros", "metal"] }

and

candle-core = { git = "https://github.com/EricLBuehler/candle.git", version = "0.6.0", rev = "c967be9", features=["metal"] }