[install-help]: Is it possible to change the default model?

[aef5748]

If I want to change the default model dolphin-2.2.1-mistral-7b.Q5_K_M.gguf to another model like as Meta-Llama-3.1-8B-Instruct.Q4_K_M.gguf or a custom-defined model, how should I modify it?

[kyteinsky]

hello,
I'd be happy to explain. The config contains many options but for this one, the llm option is important. All the listed items there llama, hugging_face, etc. are examples of configs for the LLM model to load/the LLM backend to use. Right now only the first one is used. That should be llama for you but the new config allows to use the TaskProcessing tasks for the answer generation too (nc_texttotext).

Now, to answer the real question,

1. place the GGUF model file inside /nc_app_context_chat_backend_data/ inside the docker container nc_app_context_chat_backend. Use this command for that:

docker cp <path_to_gguf_file> nc_app_context_chat_backend:/nc_app_context_chat_backend_data/

2. modify the config file (/nc_app_context_chat_backend_data/config.yaml) and set the new model's filename at llm->llama->model_path
3. restart the docker container

docker restart nc_app_context_chat_backend

For llama, you can see all the available options here: https://api.python.langchain.com/en/latest/llms/langchain_community.llms.llamacpp.LlamaCpp.html