Fix broken NeMo dependencies

[sarahyurick]

See example failure: https://github.com/NVIDIA/NeMo-Curator/actions/runs/11844241061/job/33007266564?pr=318

[sarahyurick]

What do we think about this?

ModuleNotFoundError: No module named 'nemo'

[ryantwolf]

Based on our discussions from slack, I think we can just transform this class to be something like this:

class TokenizerFertilityFilter(DocumentFilter):

    def __init__(self, path_to_tokenizer=None, min_char_to_token_ratio=2.5):
        if path_to_tokenizer is None:
            raise ValueError(
                "Must provide a valid path to a SentencePiece " "tokenizer"
            )
        self._tokenizer = sentencepiece.SentencePieceProcessor()
        self._tokenizer.Load(path_to_tokenizer)
        self._threshold = min_char_to_token_ratio

        self._name = "tokenizer_fertility"

    def score_document(self, source):
        tokens = self._tokenizer.encode_as_pieces(source)
        num_chars = len(source)
        num_tokens = len(tokens)
        if num_tokens == 0:
            return -1
        return num_chars / num_tokens

    def keep_document(self, score):
        return score >= self._threshold

Then we can just delete the one file you copied over. Lmk what you think.

[VibhuJawa]

We should probably run this via a batch instead of running it on a per file pieces and return a single file. We can also probably use crossfit for it (if we want to)

[VibhuJawa]

This is what that will look like

 cf.op.Tokenizer(model, cols=["text"], tokenizer_type="sentencepiece")

That said, we might have to ensure this works in a CPU environment too so there might be some complexity here we need to fix.

[sarahyurick]

Thanks @VibhuJawa ! I have opened #377 to track this.

[sarahyurick]

Copied from https://github.com/NVIDIA/NeMo/blob/main/nemo/export/sentencepiece_tokenizer.py

[sarahyurick]

Opened #376.

[ryantwolf]

We also probably need to add torch as a dependency now. We inherited that from NeMo. Though, not sure if the HF libraries pick that up automatically.

[sarahyurick]

We also probably need to add torch as a dependency now. We inherited that from NeMo. Though, not sure if the HF libraries pick that up automatically.

Looking at the logs, it looks like torch is being pulled in by some other library already.

Link: https://productionresultssa9.blob.core.windows.net/actions-results/ce844066-1bf6-404f-895b-7e1ac7432dc9/workflow-job-run-027ca1db-8b7e-50a2-b77c-248d223202be/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-11-15T22%3A43%3A41Z&sig=RSceDRbNmBHo%2FCRO2vfSL6PONPAkSB7BPoNJdUUYwU8%3D&ske=2024-11-16T07%3A52%3A48Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2024-11-15T19%3A52%3A48Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2024-08-04&sp=r&spr=https&sr=b&st=2024-11-15T22%3A33%3A36Z&sv=2024-08-04

[ryantwolf]

@sarahyurick I think there's a place in the user guide under images/gettingstarted.rst that has the cython install instructions too.

[sarahyurick]

@sarahyurick I think there's a place in the user guide under images/gettingstarted.rst that has the cython install instructions too.

Yes I have updated docs/user-guide/image/gettingstarted.rst - let me know if there's another somewhere.

[ryantwolf]

oops sorry I was blind. I'm used to the user guide being the first thing in the side bar. Looks good.