Not building with CUBLAS in Windows?

[pinballelectronica]

Hi. I am trying to compile this as a dependency on another project and noticed that the env settings don't seem to be taking. I'm not sure if this is supposed to work on Windows or not so maybe that's the issue. I'll try WSL next and failing that, Docker.

EX:

F:\privateGPT>set LLAMA_CUBLAS=1 && set FORCE_CMAKE=1 && pip install llama-cpp-python

Wheel builds but when running it doesn't seem to be using CUDA (0% GPU). No errors.

F:\privateGPT>python --version
Python 3.10.10

also:

i9-12900 128g
4090
cuda 12.0

Thanks!

[abetlen]

@pinballelectronica can you try the new instructions in the README, some users mentioned using CMAKE_ARGS works more consistently.

[CapitalBeyond]

I have made a powershell cublas compile script for windows. It is under issues and it works for me.

[pinballelectronica]

I'm hearing there isn't even much of a performance improvement but will try anyway.

[gjmulder]

I'm seeing 50% YMMV

[pinballelectronica]

@gjmulder that's nothing to discount. will look. thanks.