All my custom dataset examples are skipped.

[dinoelT]

I have a finetuned the llama 7b model to output a json structure given a raw document.
It works pretty well, but I wanted to try to make it faster by quantizing it.
In the calib_data.py file, the following lines cause all the examples I have to be skipped.
for data in dataset:
line = data[text_column]
line = line.strip()
line_encoded = tokenizer.encode(line)
if len(line_encoded) > 512:
continue
Could you tell me why we check the number of tokens(is it required?) and what could I do to solve this problem?
Maybe split my example into two parts with overlap?

Thank you