v0.1.0
Though v0.1.0, it is a alpha version with features as bellow.
- build with
tokenizers-cpp(a cpp binding for rust code) for tokenizer- support token normalize
- support CJK and so on
- use gguf format
- support special token
- support eps
Though v0.1.0, it is a alpha version with features as bellow.
tokenizers-cpp (a cpp binding for rust code) for tokenizer