is max_seq_len configurable or hardcoded parameter?

[morphpiece]

Today I was going to train a gpt3_124m model, when I noticed that the max_seq_len is hardcoded here and at the same time, it's a configurable parameter here. Then I executed the training executable with and without -t 2048 and find that the number of model parameters stays the same at 124475904.

is there a 🐛 bounty? 😃

[gordicaleksa]

yeah, we're focused on using the exact setup of gpt-2/3 right now, the idea is not to make this as configurable as possible, many things are hardcoded which goes against the general SWE principles, but it's less important given the goal of this project.

[morphpiece]

Two points come to mind.

It's fine to hardcode at this stage (like vocabulary), but then it should not be available as a configurable parameter. That can throw someone off completely. It's better to fix it like vocabulary is fixed and not be available as a configurable input.

Secondly, since one of the usecases is GPT-3 (which uses max_seq_len of 2048), it is assumed that users will change this value from the hardcoded 1024 to 2048. In that respect, probably hardcoding this param needs to be reconsidered.

Thanks again for this amazing work!