Difference between AutoTiktokenizer and HF AutoTokenizer

[amogkam]

Seems that AutoTiktokenizer is handling newline characters differently than Huggingface.

from autotiktokenizer import AutoTikTokenizer
from transformers import AutoTokenizer

text = "This is an \nexample"

hf_tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
auto_tokenizer = AutoTikTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

hf_tokens = hf_tokenizer.encode(text, add_special_tokens=False)
auto_tokens = auto_tokenizer.encode(text)

assert hf_tokens == auto_tokens # This fails.