Different result between Llama-cpp-python and Llama-cpp

[AndreCarasas]

Hi, I'm wondering if someone has the same problem when using Llama-cpp-python comparing it to Llama.cpp, in my case this is the result i'm getting:

Llama.cpp

/ Llama.cpp code
./main --interactive-first \ -t 8 \ -ngl 1 \ -m ./models/13b/llama-2-13b-chat.Q5_K_M.gguf \ --color \ -c 2048 \ --temp 0.7 \ --repeat_penalty 1.1 \ -s 43 \ -n -1 \ --multiline-input

Llama.cpp prompt

Using this format: Category--Explanation (USING DASHES TO SEPARATE)
Assign this review to one of the following categories: A. Inventory, B. Customer Service, or C. Product. Don't rewrite the Example, just give me the category and briefly explain why: ’I had a terrible experience at this restaurant. The staff was rude and unhelpful, and the food was mediocre at best. I would not recommend this place to anyone.’

Llama.cpp Answer

"Answer: C. Product - This review is focused on the quality of the food and the reviewer's negative experience with it, so it falls under the category of "Product"."

Llama-cpp-python

Llama-cpp-python code

from llama_cpp import Llama
import pandas as pd
params = {
    'model_path': "./models/13b/llama-2-13b-chat.Q5_K_M.gguf",
    'n_ctx': 2048,
    'seed': 43,
    'n_threads': 8,
    #'n_batch': 256,
    'n_gpu_layers': 1
}

output = llm("\
        Using this format: Category--Explanation (USING DASHES TO SEPARATE) \
        Assign this review to one of the following categories: A. Inventory, B. Customer Service, or C. Product. \
            Don't rewrite the Example, just give me the category and briefly explain why: \
            ’I had a terrible experience at this restaurant. The staff was rude and unhelpful, and the food was mediocre at best. I would not recommend this place to anyone.’ 
",temperature=0.7,repeat_penalty=1.1,echo=False)
print(output)

The result on Python is :

"
{'id': 'cmpl-7d490a72-425a-491e-86a8-9be7dbe6ed01', 'object': 'text_completion', 'created': 1696801394, 'model': './models/13b/llama-2-13b-chat.Q5_K_M.gguf', 'choices': [{'text': " What category would you assign this review? A. Inventory (staff) B. Customer Service (staff and food) C. Product (food) This is a classic example of a negative review that covers multiple categories, so it would be difficult to assign it to just one. However, if I had to choose, I would say that Category B (Customer Service) would be the most appropriate, since the reviewer's primary complaints are about the staff's behavior and the quality of the food.\n\nAnswer: B. Customer Service", 'index': 0, 'logprobs': None, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 107, 'completion_tokens': 117, 'total_tokens': 224}}
"

As you can see the answers using LLama.cpp and Llama-cpp-python are different even when they use the same seed and the python one seems to be not as precise as the Llama.cpp, Any idea if there is something that I can do to replicate the Llama.cpp answer using python?

Thanks in advance!

[kkkkkkjd]

I have the same confusion as well.

[pikaro]

Your code is incomplete. For one thing, llm would be undefined, it's not clear what you're calling. And params seems to be unused - you'd have to splat it into the llm call with **params for it to have any effect.

[hosseinalipour]

when installing llama-cpp-python it compile its own llama.cpp. use the same version of llama.cpp and you will get the same result. you might want to use this