Ngram instead of paragraph removal?

[IanMagnusson]

Hi @dirkgr! Here is a feature that would be very much desirable for decontamination, but I'm not sure how difficult it would be to implement into BFF:

The essential part of the feature would be to remove ngrams instead of paragraphs. This is an approach used by GPT3 and TNLG and other than minHash based decontamination it's the other decontamination approach that has actually been used on LLMs in the literature.

Here is the exact specification from TNLG: "We use n-grams to remove texts that occur in the downstream tasks from the training datasets. When we find an n-gram match between a task document and a training document, we split the training document into two pieces by removing the n-gram along with 200 characters from both of its sides. We also remove any split training document with fewer than 200 characters, or training documents which were split more than 10 times."

I wonder if BFF's bloom filter would be necessary to store all the ngrams from the in a several million document combined eval set? If not I should probably consider building this into the WIMBD tool instead.

Thanks for any thoughts you have on this!

[IanMagnusson]

@dirkgr and I spoke about this and refined the specification somewhat:

• a new option would enable the tool to check ngrams over the whole document rather than paragraph by paragraph
• it would also record the raw count of ngrams from each document found in the bloom filter