fixed num_workers

[d-kleine]

• removed num_workers in ch04/01 for GPTDatasetV1(Dataset) and added to DataLoader()
• changed num_workers=0 in DataLoader() to num_workers=num_workers as defined in create_dataloader_v1()
• added num_workers to create_dataloader_v1 to ch06 and ch07

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[d-kleine]

Please double check that everything is fine 🙂

[rasbt]

Oh thanks, this wasn't supposed to be hardcoded, otherwise it would defeat the purpose of the function argument!

[rasbt]

Btw, in case you tried, how is the data loading with multiple workers for you on Windows? I remember a few years ago there were issues with that since Windows had some limitations with multiprocessing in Python. Not sure if something has changed since then.

[d-kleine]

Haven't tested it yet, but just tried. With the 124M model, I can see no big difference:

[rasbt]

Oh yeah, you might only notice if you train on GPU. But text loading is so cheap, especially because we pretokenize, that there will probably be only a very very small difference. Thanks for checking though, seems like it generally works now on Windows without crashing :)

[d-kleine]

and here for GPT-2 XL:
(edit: updated):

[rasbt]

I think the bottom one uses the small model still. It's also more the training time that would be affected (not the creation of the data loaders).

[d-kleine]

Thanks, screenshot updated above ⬆️

BTW I just see as that the dropout for the gpt-2 models changes between 0 and 0.1 over the chapters, is this intended?

[rasbt]

BTW I just see as that the dropout for the gpt-2 models changes between 0 and 0.1 over the chapters, is this intended

Yes, in chapter 4 we use dropout 0.1 because that's the original setting OpenAI used in the GPT-2 paper. But nowadays it's not necessary/recommended to use dropout, so I am not using it during finetuning.

[d-kleine]

Please check ch05/03_bonus_pretraining_on_gutenberg/pretraining_simple.py. It's not technically relevant, but shouldn't the dropout be the same for the small debugging model as for the standard model ("drop_rate": 0.1)?

(...) so I am not using it during finetuning.

Ah, I see, thanks! I think it would be good to add this info in the text or at least to the code as a code comment where you do "drop_rate": 0.0, adding a comment like

BASE_CONFIG = {
    ...
    "drop_rate": 0.0, # deactivated as dropout in LLMs is not recommended anymore
    ...

[d-kleine]

I just have added num_workers to create_dataloader_v1 to ch06 and ch07 too as it makes the previous_chapters.py files more consistent

About the PR, is everything fine so far about the changes?

[rasbt]

About the PR, is everything fine so far about the changes?

Yes, this looks awesome, many thanks!

[d-kleine]

Have now tested the num_workers in the training phase as well, only small, negligible improvements with same batch size

[rasbt]

Interesting. Yeah, I think that's the same what I originally observed on Linux + GPU (and also macOS on CPU). I hypothesize because the data is pretokenized, loading is quick no matter what. Thanks for looking into it!