[Bug] [model] Fail to load model with llama.cpp

[cestbonn]

Search before asking

• I had searched in the issues and found no similar issues.

Operating system information

MacOS(M1, M2...)

Python version information

=3.11

DB-GPT version

main

Related scenes

• Chat Data
• Chat Excel
• Chat DB
• Chat Knowledge
• Model Management
• Dashboard
• Plugins

Installation Information

• Installation From Source

• Docker Installation

• Docker Compose Installation

• Cluster Installation

• AutoDL Image

• Other

Device information

Device: M2

Models information

LLM: ggml-model-q4_0.bin
embedding: large-chinese

What happened

I download the model with

wget https://huggingface.co/TheBloke/vicuna-7B-v1.5-GGML/resolve/main/vicuna-7b-v1.5.ggmlv3.q4_K_M.bin -O models/ggml-model-q4_0.bin

and rewrite the .env file, but get this error:

2023-10-02 19:45:30 xxxMacBook-Air.local pilot.model.llm.llama_cpp.llama_cpp[2889] INFO Cache capacity is 0 bytes
2023-10-02 19:45:30 xxxMacBook-Air.local pilot.model.llm.llama_cpp.llama_cpp[2889] INFO Load LLama model with params: {'model_path': '/xxx/DB-GPT/models/ggml-model-q4_0.bin', 'n_ctx': 4096, 'seed': -1, 'n_threads': None, 'n_batch': 512, 'use_mmap': True, 'use_mlock': False, 'low_vram': False, 'n_gpu_layers': 1000000000, 'n_gqa': None, 'logits_all': True, 'rms_norm_eps': 5e-06}
gguf_init_from_file: invalid magic number 67676a74
error loading model: llama_model_loader: failed to load model from /xxx/DB-GPT/models/ggml-model-q4_0.bin

llama_load_model_from_file: failed to load model
2023-10-02 19:45:30 xxxMacBook-Air.local pilot.model.cluster.worker.default_worker[2889] WARNING Model has been stopped!!

What you expected to happen

I follow the installation step by step but get this error

How to reproduce

python pilot/server/dbgpt_server.py

Additional context

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[cestbonn]

llama.cpp stoped supporting for ggml model since ver 0.1.79, so use llama-cpp-python==0.1.78 or gguf model. :D

[fangyinc]

llama.cpp stoped supporting for ggml model since ver 0.1.79, so use llama-cpp-python==0.1.78 or gguf model. :D

Aha, see here