Does llama.cpp server ready for production use?

[bash99]

I've try it with Qwen 2.5 32B/72B, perfect for single turn Q&A, but in multi-turn conversation, it got some bugs, like core dumped, or after normal output for a while, it start output random strings, some times it end, sometimes not.

For "random strings output bug", I can make a repeatable case, just wonder is it worth?

I'm use Open Web UI as frontend, and the bug won't occurs when switch to aphrodite-engine.

I've try K/V cache with default(f16) q8_0 q4_0, all of them had "random strings output bug", the later two had "core dumped bug"

[ggerganov]

Likely a misconfiguration issue. I'm using Qwen 2.5 32B daily without any issues.

[bash99]

I don't know too many on config? or I'm make it wrong?

I start it with

CUDA_VISIBLE_DEVICES=0,1 ~/bin/llama.cpp/llama-server -a Qwen2.5-72B-Instruct -m ./qwen2.5-72b-instruct-q4_k_m.gguf -np 20 -ngl 240 -c 16384  -fa -ctk q8_0 -ctv q8_0 --host 0.0.0.0 --port 7865 \
-p ~/bin/llama.cpp/prompts/chat-with-qwen.txt

I'm using it on V100-32G * 2, llama.cpp(rev ccb6809) is compiled with cuda/nvcc 12.1.

[ggerganov]

Your per-slot context is only 16384/20 ~ 800 tokens. Try this instead: -c 16384 -np 1

[bash99]

Your per-slot context is only 16384/20 ~ 800 tokens. Try this instead: -c 16384 -np 1

So it's a static config instead of dynamic config?
for example if there are 20 simple concurrent requests, I hope each request's context is 800 tokens.

But if there are only one long context request, I hope all context would be assigned to it. I think vllm have some similar features?

[ExtReMLapin]

Is your binary up to date, few weeks ago crash was linked to kv cache quantization + defrag