Am I being limited by single core CPU performance when fully offloaded?

[reversebias]

I'm using a system with the following hardware:

• Xeon W-2133
• 96GB DDR4-2666
• 2x Nvidia P40 each in a 16x 3.0 slot

Running a Q6 quant of mixtral with:
./main -m ~/models/mixtral-8x7b-instruct-v0.1.Q6_K.gguf -c 32768 -ngl 128 -ts 39,61,0 -sm row -t 8 --prompt "Once upon a time"

And getting very decent 21-22 t/s:
llama_print_timings: sample time = 155.18 ms / 352 runs ( 0.44 ms per token, 2268.29 tokens per second) llama_print_timings: prompt eval time = 168.00 ms / 5 tokens ( 33.60 ms per token, 29.76 tokens per second) llama_print_timings: eval time = 16486.27 ms / 351 runs ( 46.97 ms per token, 21.29 tokens per second) llama_print_timings: total time = 16912.39 ms / 356 tokens

However I noticed that during generation, there's a single CPU thread pegged at 100%:

Single threaded is expected here, from looking at commits like #5238

But with the single thread at 100%, and nvidia-smi showing 50-60% utilization:

Could my single thread performance be my bottleneck?

[reversebias]

Found a response here that suggests it's spinning a thread and not doing anything, so not the bottleneck: #3210 (comment)

[ggerganov]

The thread spinning referenced in the comment is no longer occurring with latest ggml-backend updates

[ExtReMLapin]

I feel like i'm having the same issue

[Artefact2]

Similar behaviour here on AMD/rocBLAS.

Most of the CPU time is spent in rocr::core::InterruptSignal::WaitRelaxed(hsa_signal_condition_t, long, unsigned long, hsa_wait_state_t). Needless to say, this makes the CPU generate a lot of heat and consume extra power for nothing.

[moritzschaefer]

I would have suspected that more CPUs should at least benefit initial tokenization (is that the load part of the timings?).

[jwhitehorn]

I too am running into this behavior. Single CPU thread at 100%, and GPU under-utilized (about 20% utilization).

Not sure if it matters, but here are some details:

• Debian 12 / 6.8.4 host
• Dual Xeon E5 2697v2 CPUs
• 64GB ECC RAM (Quad-channel DDR3-1333)
• Intel Arc A770 GPU
• Llama.cpp (via llama-cpp-python 0.2.65) dockerized using the intel/oneapi-basekit:2024.1.0-devel-ubuntu22.04 image
• I've tried different models (llama 2, llama 3, claude 2, etc), all fully offloaded to VRAM

I've tried using llama.cpp's main and server executables directly and can confirm that the fact that I'm running through Python seems immaterial to the issue.

If the CPU thread isn't the bottleneck, do we know why the GPU isn't more utilized?

[fidecastro]

I'm also seeing similar behavior on my system. My situation might make it a bit more interesting a problem since I wasn't expecting to see both a single CPU thread and GPU maxing out, as I'm seeing here.

Currently running the latest llama.cpp release (b3484) on the following machine:

2x Xeon E5-2690v4 (total 28 cores, 56 threads)
128 GB DDR4 RAM 2133
4 x NVIDIA P40 (3 on 16x PCIe 3.0, 1 on 8x PCIe 3.0)

I compiled llama.cpp with CUDA support on a Docker container based on this Docker image (Ubuntu 22.04 with CUDA 12.5.1 installed), and which is running with full access to all host GPUs and CPU cores. I also installed flash-attention and am running q4_K_M llama3.1-70B-Instruct.

I'm calling llama-cli with the following terminal command: ./llama-cli -m /workspace/llama3.1/Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf -ctk q4_0 -ctv q4_0 -fa -ngl 999 -e --chat-template llama3 -n 8000 --file prompt.txt

HTOP tells me there's a single CPU with 100% utilization, while my GPUs cycle between idle (3 of the 4 GPUs are always like that) and 100% use (only 1 GPU is in use at a given time).

[slaren]

You may get better results with -sm row with the P40.

[stevenbedrick]

Just chiming in that I am also seeing the same behavior! Model fully offloaded across multiple GPUs (A40s in my case), and then during generation I'm seeing a single CPU thread, pegged at 100, with the GPUs being underutilized. I've verified that the behavior persists regardless of split mode. 🤷‍♂️

[8XXD8]

I have the same problem, with 4X AMD GPUs, one thread maxed out, and low GPU utilization.
For me, --no-mmap helps.

[stevenbedrick]

Hmm, thanks; in my case that doesn't seem to make much of a difference. 🤷‍♂️

[ClonedPoro]

Is there any updates on this?/ Have you found a solution or workaround? Im seeing the exact same behaviour with a fully offloaded model (still got dedicated memory to spare on the GPU) and barely any GPU utilization but 100% on CPU core 4 in my case (rest is mostly idle).
During initilization my model reads:
AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |

[AGenchev]

My experimets: Performance on AMD APU 5600G (w BIOS GPU overclock)
I'm using same model in all tests (llama-13-v2/ggml-model-q8_0.gguf).
using Vulkan ~ 1.63 tokens per second (but strange - no load on the CPU).
using CPU - 2.86 tokens per second (around 599% load on CPU).
using RoCM HIPBLAS - 3.07 tokens per second (1 CPU core loaded at 100%).

So inference with Vulkan comes 'for free' using only the GPU. But it is slowest. Other frameworks are faster but load the CPU.

Haven't tried what it can do with OpenCL.

[KutayKoray]

Hi have you done any particular thing to use 2 gpu at the same time because i have 2 1660 super but i cant use both of them model is only recognize 1 gpu. When i serve ollama it detects both of my gpu's when i run nvidia-smi it detects both of my gpu's but when i run llama3.1 it only use one of them has any one faced with this problem or is this because of my gpu model?

[Fuckingnameless]

has always been the normal behavior for me on llamacpp when fully offloaded