New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Recommendation for decoder finetuning #61

Open

elias-ramzi opened this issue Sep 19, 2024 · 0 comments

elias-ramzi commented Sep 19, 2024 •

edited

Loading

Hi,

I am trying to finetune the decoder only of the tokenizer on a new dataset. I was wondering if you could share some finetuning recipies.

Mainly:

Should the discriminator loss be used / finetuned at the same time on the new dataset
Should I activate the discriminator loss directly at the begining of training
Should I keep the same hyperparameters as training (lr=1e-4, weight_decay=0.05)
Should I use the optimizer state of the pretraining

Thank you for your help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment