The proper way to activate cuBLAS via CMD on a windows machine

[andyjh86]

Hello,

I've seen a lot of people having issues with installing with the cuBLAS flags active via CMD on windows machines.

You have to set them all via the same line while linking them with &&

For example

SET LLAMA_CLBLAST=1 && SET CMAKE_ARGS="-DLLAMA_CLBLAST=on" && SET FORCE_CMAKE=1 && python -m pip install llama-cpp-python

or to reinstall

SET LLAMA_CLBLAST=1 && SET FORCE_CMAKE=1 && SET CMAKE_ARGS=-DLLAMA_CUBLAS=on && python -m pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir

This will get cuBLAS working if you're doing it via CMD.

For what ever reason, setting the flags one at a time won't work. There are a few fixes I've seen using a .bat file which pretty much does the same way but it's a lot easier just to set the flags via CMD and install it via PIP in the usual way.

Also, the -m flag isn't necessary when using python pip but is best practice in my opinion which is why I've added it in the examples :-)

Maybe this should be added to the main page instructions for windows machines?

Thanks

[gsscoder]

Hi, thanks for sharing this but it doesn't work for me :(

The installation ends with a multiple syntax errors and macro expansion issues

Python: 3.11.4
OS: Windows 11

This is the complete command output:

SET LLAMA_CLBLAST=1 && SET FORCE_CMAKE=1 && SET CMAKE_ARGS=-DLLAMA_CUBLAS=on && python -m pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir
Collecting llama-cpp-python
  Downloading llama_cpp_python-0.1.79.tar.gz (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 5.3 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting typing-extensions>=4.5.0 (from llama-cpp-python)
  Obtaining dependency information for typing-extensions>=4.5.0 from https://files.pythonhosted.org/packages/ec/6b/63cc3df74987c36fe26157ee12e09e8f9db4de771e0f3404263117e75b95/typing_extensions-4.7.1-py3-none-any.whl.metadata
  Downloading typing_extensions-4.7.1-py3-none-any.whl.metadata (3.1 kB)
Collecting numpy>=1.20.0 (from llama-cpp-python)
  Obtaining dependency information for numpy>=1.20.0 from https://files.pythonhosted.org/packages/72/b2/02770e60c4e2f7e158d923ab0dea4e9f146a2dbf267fec6d8dc61d475689/numpy-1.25.2-cp311-cp311-win_amd64.whl.metadata
  Downloading numpy-1.25.2-cp311-cp311-win_amd64.whl.metadata (5.7 kB)
Collecting diskcache>=5.6.1 (from llama-cpp-python)
  Downloading diskcache-5.6.1-py3-none-any.whl (45 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.6/45.6 kB ? eta 0:00:00
Downloading numpy-1.25.2-cp311-cp311-win_amd64.whl (15.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.5/15.5 MB 7.5 MB/s eta 0:00:00
Downloading typing_extensions-4.7.1-py3-none-any.whl (33 kB)
Building wheels for collected packages: llama-cpp-python
  Building wheel for llama-cpp-python (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for llama-cpp-python (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [168 lines of output]


      --------------------------------------------------------------------------------
      -- Trying 'Ninja (Visual Studio 17 2022 x64 v143)' generator
      --------------------------------
      ---------------------------
      ----------------------
      -----------------
      ------------
      -------
      --
      Not searching for unused variables given on the command line.
      CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
        Compatibility with CMake < 3.5 will be removed from a future version of
        CMake.

        Update the VERSION argument <min> value or use a ...<max> suffix to tell
        CMake that the project does not need compatibility with older versions.


      -- The C compiler identification is MSVC 19.36.32537.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.36.32532/bin/Hostx86/x64/cl.exe - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- The CXX compiler identification is MSVC 19.36.32537.0
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.36.32532/bin/Hostx86/x64/cl.exe - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Configuring done (1.9s)
      -- Generating done (0.0s)
      -- Build files have been written to: C:/Users/Coder/AppData/Local/Temp/pip-install-3c8wc5ry/llama-cpp-python_4efadcce9ae74c2f85b8c6008955f4e0/_cmake_test_compile/build
      --
      -------
      ------------
      -----------------
      ----------------------
      ---------------------------
      --------------------------------
      -- Trying 'Ninja (Visual Studio 17 2022 x64 v143)' generator - success
      --------------------------------------------------------------------------------

      Configuring Project
        Working directory:
          C:\Users\Coder\AppData\Local\Temp\pip-install-3c8wc5ry\llama-cpp-python_4efadcce9ae74c2f85b8c6008955f4e0\_skbuild\win-amd64-3.11\cmake-build
        Command:
          'C:\Users\Coder\AppData\Local\Temp\pip-build-env-88h2yqx_\overlay\Lib\site-packages\cmake\data\bin/cmake.exe' 'C:\Users\Coder\AppData\Local\Temp\pip-install-3c8wc5ry\llama-cpp-python_4efadcce9ae74c2f85b8c6008955f4e0' -G Ninja '-DCMAKE_MAKE_PROGRAM:FILEPATH=C:\Users\Coder\AppData\Local\Temp\pip-build-env-88h2yqx_\overlay\Lib\site-packages\ninja\data\bin\ninja' -D_SKBUILD_FORCE_MSVC=1930 --no-warn-unused-cli '-DCMAKE_INSTALL_PREFIX:PATH=C:\Users\Coder\AppData\Local\Temp\pip-install-3c8wc5ry\llama-cpp-python_4efadcce9ae74c2f85b8c6008955f4e0\_skbuild\win-amd64-3.11\cmake-install' -DPYTHON_VERSION_STRING:STRING=3.11.4 -DSKBUILD:INTERNAL=TRUE '-DCMAKE_MODULE_PATH:PATH=C:\Users\Coder\AppData\Local\Temp\pip-build-env-88h2yqx_\overlay\Lib\site-packages\skbuild\resources\cmake' '-DPYTHON_EXECUTABLE:PATH=C:\Users\Coder\miniconda3\envs\localGPT\python.exe' '-DPYTHON_INCLUDE_DIR:PATH=C:\Users\Coder\miniconda3\envs\localGPT\Include' '-DPYTHON_LIBRARY:PATH=C:\Users\Coder\miniconda3\envs\localGPT\libs\python311.lib' '-DPython_EXECUTABLE:PATH=C:\Users\Coder\miniconda3\envs\localGPT\python.exe' '-DPython_ROOT_DIR:PATH=C:\Users\Coder\miniconda3\envs\localGPT' -DPython_FIND_REGISTRY:STRING=NEVER '-DPython_INCLUDE_DIR:PATH=C:\Users\Coder\miniconda3\envs\localGPT\Include' '-DPython_LIBRARY:PATH=C:\Users\Coder\miniconda3\envs\localGPT\libs\python311.lib' '-DPython3_EXECUTABLE:PATH=C:\Users\Coder\miniconda3\envs\localGPT\python.exe' '-DPython3_ROOT_DIR:PATH=C:\Users\Coder\miniconda3\envs\localGPT' -DPython3_FIND_REGISTRY:STRING=NEVER '-DPython3_INCLUDE_DIR:PATH=C:\Users\Coder\miniconda3\envs\localGPT\Include' '-DPython3_LIBRARY:PATH=C:\Users\Coder\miniconda3\envs\localGPT\libs\python311.lib' '-DCMAKE_MAKE_PROGRAM:FILEPATH=C:\Users\Coder\AppData\Local\Temp\pip-build-env-88h2yqx_\overlay\Lib\site-packages\ninja\data\bin\ninja' -DLLAMA_CUBLAS=on -DCMAKE_BUILD_TYPE:STRING=Release -DLLAMA_CUBLAS=on

      Not searching for unused variables given on the command line.
      -- The C compiler identification is MSVC 19.36.32537.0
      -- The CXX compiler identification is MSVC 19.36.32537.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.36.32532/bin/Hostx86/x64/cl.exe - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.36.32532/bin/Hostx86/x64/cl.exe - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Found Git: C:/Program Files/Git/cmd/git.exe (found version "2.41.0.windows.1")
      fatal: not a git repository (or any of the parent directories): .git
      fatal: not a git repository (or any of the parent directories): .git
      CMake Warning at vendor/llama.cpp/CMakeLists.txt:118 (message):
        Git repository not found; to enable automatic generation of build info,
        make sure Git is installed and the project is a Git repository.


      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
      -- Looking for pthread_create in pthreads
      -- Looking for pthread_create in pthreads - not found
      -- Looking for pthread_create in pthread
      -- Looking for pthread_create in pthread - not found
      -- Found Threads: TRUE
      -- Could not find nvcc, please set CUDAToolkit_ROOT.
      CMake Warning at vendor/llama.cpp/CMakeLists.txt:292 (message):
        cuBLAS not found


      -- CMAKE_SYSTEM_PROCESSOR: AMD64
      -- x86 detected
      -- Configuring done (2.3s)
      -- Generating done (0.0s)
      -- Build files have been written to: C:/Users/Coder/AppData/Local/Temp/pip-install-3c8wc5ry/llama-cpp-python_4efadcce9ae74c2f85b8c6008955f4e0/_skbuild/win-amd64-3.11/cmake-build
      [1/11] Building C object vendor\llama.cpp\CMakeFiles\ggml.dir\ggml-alloc.c.obj
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\stdio.h(378): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\stdio.h(2437): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\corecrt_search.h(188): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\stdlib.h(79): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\stdlib.h(1286): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\corecrt_memory.h(76): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\corecrt_wstring.h(573): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\string.h(531): warning C5105: macro expansion producing 'defined' has undefined behavior
      [2/11] Building C object vendor\llama.cpp\CMakeFiles\ggml.dir\k_quants.c.obj
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\corecrt_math.h(44): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\corecrt_math.h(963): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\corecrt_memory.h(76): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\corecrt_wstring.h(573): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\string.h(531): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\malloc.h(173): warning C5105: macro expansion producing 'defined' has undefined behavior
      [3/11] Building C object vendor\llama.cpp\CMakeFiles\ggml.dir\ggml.c.obj
      FAILED: vendor/llama.cpp/CMakeFiles/ggml.dir/ggml.c.obj
      C:\PROGRA~1\MICROS~3\2022\COMMUN~1\VC\Tools\MSVC\1436~1.325\bin\Hostx86\x64\cl.exe  /nologo -DGGML_USE_K_QUANTS -D_CRT_SECURE_NO_WARNINGS -IC:\Users\Coder\AppData\Local\Temp\pip-install-3c8wc5ry\llama-cpp-python_4efadcce9ae74c2f85b8c6008955f4e0\vendor\llama.cpp\. /DWIN32 /D_WINDOWS /O2 /Ob2 /DNDEBUG -std:c11 -MD /arch:AVX2 /showIncludes /Fovendor\llama.cpp\CMakeFiles\ggml.dir\ggml.c.obj /Fdvendor\llama.cpp\CMakeFiles\ggml.dir\ /FS -c C:\Users\Coder\AppData\Local\Temp\pip-install-3c8wc5ry\llama-cpp-python_4efadcce9ae74c2f85b8c6008955f4e0\vendor\llama.cpp\ggml.c
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\malloc.h(173): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\time.h(589): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\corecrt_math.h(44): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\corecrt_math.h(963): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\corecrt_search.h(188): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\stdlib.h(79): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\stdlib.h(1286): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\corecrt_memory.h(76): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\corecrt_wstring.h(573): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\string.h(531): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\stdio.h(378): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\stdio.h(2437): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\float.h(328): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\ctype.h(241): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\\include\10.0.17763.0\\um\winbase.h(9254): warning C5105: macro expansion producing 'defined' has undefined behavior
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um\oaidl.h(487): warning C5103: pasting '/' and '/' does not result in a valid preprocessing token
      C:\Program Files (x86)\Windows Kits\10\\include\10.0.17763.0\\shared\wtypes.h(745): note: in expansion of macro '_VARIANT_BOOL'
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um\oaidl.h(487): error C2059: syntax error: '/'
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um\oaidl.h(502): warning C5103: pasting '/' and '/' does not result in a valid preprocessing token
      C:\Program Files (x86)\Windows Kits\10\\include\10.0.17763.0\\shared\wtypes.h(745): note: in expansion of macro '_VARIANT_BOOL'
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um\oaidl.h(502): error C2059: syntax error: '/'
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um\oaidl.h(530): error C2059: syntax error: '}'
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um\oaidl.h(531): error C2059: syntax error: '}'
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um\oaidl.h(533): error C2059: syntax error: '}'
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um\oaidl.h(534): error C2059: syntax error: '}'
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um\oaidl.h(665): error C2079: 'varDefaultValue' uses undefined struct 'tagVARIANT'
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um\oaidl.h(950): error C2079: 'varValue' uses undefined struct 'tagVARIANT'
      C:\Program Files (x86)\Windows Kits\10\\include\10.0.17763.0\\um\propidlbase.h(319): warning C5103: pasting '/' and '/' does not result in a valid preprocessing token
      C:\Program Files (x86)\Windows Kits\10\\include\10.0.17763.0\\shared\wtypes.h(745): note: in expansion of macro '_VARIANT_BOOL'
      C:\Program Files (x86)\Windows Kits\10\\include\10.0.17763.0\\um\propidlbase.h(319): error C2059: syntax error: '/'
      C:\Program Files (x86)\Windows Kits\10\\include\10.0.17763.0\\um\propidlbase.h(378): error C2371: 'pvarVal': redefinition; different basic types
      C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um\oaidl.h(510): note: see declaration of 'pvarVal'
      C:\Program Files (x86)\Windows Kits\10\\include\10.0.17763.0\\um\propidlbase.h(379): error C2059: syntax error: '}'
      C:\Program Files (x86)\Windows Kits\10\\include\10.0.17763.0\\um\propidlbase.h(380): error C2059: syntax error: '}'
      C:\Program Files (x86)\Windows Kits\10\\include\10.0.17763.0\\um\propidlbase.h(383): error C2059: syntax error: '}'
      C:\Program Files (x86)\Windows Kits\10\\include\10.0.17763.0\\um\propidlbase.h(384): error C2059: syntax error: '}'
      [4/11] Building CXX object vendor\llama.cpp\common\CMakeFiles\common.dir\grammar-parser.cpp.obj
      [5/11] Building CXX object vendor\llama.cpp\common\CMakeFiles\common.dir\console.cpp.obj
      [6/11] Building CXX object vendor\llama.cpp\common\CMakeFiles\common.dir\common.cpp.obj
      [7/11] Building CXX object vendor\llama.cpp\CMakeFiles\llama.dir\llama.cpp.obj
      ninja: build stopped: subcommand failed.
      Traceback (most recent call last):
        File "C:\Users\Coder\AppData\Local\Temp\pip-build-env-88h2yqx_\overlay\Lib\site-packages\skbuild\setuptools_wrap.py", line 674, in setup
          cmkr.make(make_args, install_target=cmake_install_target, env=env)
        File "C:\Users\Coder\AppData\Local\Temp\pip-build-env-88h2yqx_\overlay\Lib\site-packages\skbuild\cmaker.py", line 697, in make
          self.make_impl(clargs=clargs, config=config, source_dir=source_dir, install_target=install_target, env=env)
        File "C:\Users\Coder\AppData\Local\Temp\pip-build-env-88h2yqx_\overlay\Lib\site-packages\skbuild\cmaker.py", line 742, in make_impl
          raise SKBuildError(msg)

      An error occurred while building with CMake.
        Command:
          'C:\Users\Coder\AppData\Local\Temp\pip-build-env-88h2yqx_\overlay\Lib\site-packages\cmake\data\bin/cmake.exe' --build . --target install --config Release --
        Install target:
          install
        Source directory:
          C:\Users\Coder\AppData\Local\Temp\pip-install-3c8wc5ry\llama-cpp-python_4efadcce9ae74c2f85b8c6008955f4e0
        Working directory:
          C:\Users\Coder\AppData\Local\Temp\pip-install-3c8wc5ry\llama-cpp-python_4efadcce9ae74c2f85b8c6008955f4e0\_skbuild\win-amd64-3.11\cmake-build
      Please check the install target is valid and see CMake's output for more information.

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for llama-cpp-python
Failed to build llama-cpp-python
ERROR: Could not build wheels for llama-cpp-python, which is required to install pyproject.toml-based projects

Any help really appreciated,
Giacomo S. S.

[tk-master]

Let me know if this guide helped #871

[bioshazard]

Is there a reason you aren't using Powershell per the README?

[andyjh86]

There was a reason but I honestly don't remember. IIRC there was issues activating the environment in powershell and setting the flags.

They'd show as flagged succesfully but it just wouldn't install with cuBLAS activated.

At the time there were a lot of people having similar issues outside GitHub so I thought I'd contribute what worked for me via CMD. People were setting the required flags one at a time, but it would install without cuBLAS activated properly despite the flags appearing to have been set correctly.
Setting all flags via && in the same line fixed the issue.