STATUS: Under development.
tiktoken is a fast BPE tokeniser for use with OpenAI's models. It provides Python interface to interact with it.
This project is a fork of original repo, bring the capability to rust world.
use tiktoken_rust as tt;
let enc = tt::get_encoding("cl100k_base").unwrap();
assert_eq!(
"hello world",
enc.decode(&enc.encode_ordinary("hello world"), tt::DecodeMode::Strict).unwrap()
)