logit-bias : Any dictionary of tokens available

[maddes8cht]

In main we have this parameter -l or --logit-bias which can be used to change the probability of certain tokens.
Is there any way to see a dictionary of the used tokens, maybe also to search them?

[KerfuffleV2]

If you specify --verbose-prompt it'll show the token ids for everything in the prompt.

Or you could look for a PyTorch format LLaMA model on HuggingFace or similar sites and check in tokenizer.json. This will have a list of all tokens in JSON format. (There's a rule against linking to LLaMA models due to legal stuff so you'd have to find that yourself if you want to browse through the list.)

[maddes8cht]

Ah, great, the prompting-method sounds like a nice way to ask for tokens.

Will the tokens be diferent for each model?
Or will they be consisten accross the Llama-Family?

[KerfuffleV2]

Or will they be consisten accross the Llama-Family?

I believe that's generally the case. I think there are some models that add an extra token at the end or something like that. These are usually special tokens like start of document, end of document, etc. Not really stuff that you'd care about manipulating via logit bias anyway.

[SlyEcho]

You have to be a little careful, because capitalization, whitespace, punctuation etc. around your words can result in different tokens. In fact llama.cpp adds a space in front of every prompt because most tokens also start with a space.