Sampler Expansion - GBNF Grammar

[ianmarmour]

This PR is functional but probably not in an ideal state? All of the grammar related code is cribbed from UtilityAI's library I'm not sure if we want to dump this and build something totally new the code seemed fairly well written and I didn't see value in re-inventing the wheel here. I will note that neither the UtilityAI or this implementation of sampler code is exactly the same as the llama.cpp sampling logic specifically we're missing the resampling behavior for increased performance when using grammars.

Additional Notes

For what it's worth I don't know if it's trivial to attempt to implement this as a sampler stage? We always need to accept the token's after sampling has been completed and a token has been selected I wasn't able to figure out a strategy for dealing with moving around the grammar in a way that made sense and it also felt awkward to include token acceptance as a SamplerStage.

Usage

Basic usage example below,

Code

let gbnf_bytes = include_bytes!("json.gbnf");
let gbnf: &str = str::from_utf8(gbnf_bytes).unwrap();
let grammar = LlamaGrammar::from_str(gbnf).unwrap();

let stages = vec![
    SamplerStage::RepetitionPenalty {
        repetition_penalty: 1.1,
        frequency_penalty: 0.0,
        presence_penalty: 0.0,
        last_n: 64,
    },
    SamplerStage::TopK(40),
    SamplerStage::TopP(0.95),
    SamplerStage::MinP(0.05),
    SamplerStage::Temperature(0.8),
];

let sampler = StandardSampler::new_softmax(stages, 1, Some(grammar));

Grammar

root ::= Response
Response ::= "{"   ws   "\"importance\":"   ws   ResponseValue   ","   ws   "\"category\":"   ws   Category   ","   ws   "\"summary\":"   ws   string   "}"
Responselist ::= "[]" | "["   ws   Response   (","   ws   Response)*   "]"
Category ::= "\"promotional\"" | "\"receipt\"" | "\"notification\"" | "'advertisement'" | "'professional'" | "'news'"
ResponseValue ::= "\"high\"" | "\"low\""
string ::= "\""   ([^"]*)   "\""
boolean ::= "true" | "false"
ws ::= [ \t\n]*
number ::= [0-9]+   "."?   [0-9]*
stringlist ::= "["   ws   "]" | "["   ws   string   (","   ws   string)*   ws   "]"
numberlist ::= "["   ws   "]" | "["   ws   string   (","   ws   number)*   ws   "]"

Output

ggml_gallocr_reserve_n: reallocating Metal buffer from size 0.00 MiB to 2114.53 MiB
ggml_gallocr_reserve_n: reallocating CPU buffer from size 0.00 MiB to 8.00 MiB
{"importance": "low", "category": "promotional", "summary": "Marketing email from Adobe Inc., most likely promoting their products and services. You can unsubscribe from future emails by clicking the link provided."}

[nkoppel]

@ianmarmour I've opened a pull request for your pull request here: ianmarmour#1