'token_embd.weight' not found with Ollama #9120
Replies: 2 comments 1 reply
-
It seems the |
Beta Was this translation helpful? Give feedback.
-
I found a workaround for this same problem. Instead using the ADAPTER in the ollama Modelfile, I used this script to merge the layers of the original model with the lora fine-tuned one: Then I converted the merged model to gguf, managed to import it into ollama and it did not cause the 'token_embd.weight' error by running it. PS: I'm not sure if this worked for me at all, because the manual tests I performed with this fine-tuned model did not turn out as I expected. |
Beta Was this translation helpful? Give feedback.
-
Hello,
I fine-tuned the Meta Llama 3.1 8B model with a HuggingFace SFTTrainer and saved the model locally. Below is the config for the trainer:
The training went well. After training I saved the model and converted it to GGUF format using llama.cpp's (latest version) script
convert_hf_to_gguf.py
with--outtype q8_0
. Then I added the model to Ollama withollama create <model-name> -f Modelfile
, same everything fine here. However when I try to run the model with the Ollama prompt I get the error :So it seems there was a problem with the conversion to GGUF and this field goes missing, am I right ? Is there anything I can do to fix it or is it a bug on llama.cpp or ollama ?
Thank you!
Beta Was this translation helpful? Give feedback.
All reactions