Replies: 1 comment
-
Hey did you make any progress on this? |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Is there a documentation of the precise algorithm of the tokenizer in llama.cpp?
While there are plenty of precise documentations or simple reference implementations for how exactly the various LLM architectures work, I can't find someting similar for (the presumably much simpler) tokenizers. But at the same time the tokenizer often seems responsible if anything breaks during port, like it happened with llama 3.
So I'm wondering if there is a documentation of what exactly llama.cpp does with tokenizer.ggml.model, tokenizer.ggml.pre, tokenizer.ggml.tokens, tokenizer.ggml.token_type, tokenizer.ggml.merges (and if some, like merges, are not present), and if there any non-trivial hard coded processing steps not governed by a parameter in the gguf.
Or do I have to read the source code?
Beta Was this translation helpful? Give feedback.
All reactions