Axolotl + ADOPT optimizer Beta1

[jwm1969]

Hey, anyone or @bursteratom if you see this, I am trying out the ADOPT version of AdamW optimizer with axolotl by specifying optimizer as adopt_adamw ... 2 questions:

1. Do we need to set decoupled=True ? If yes, how in config?
2. How do we set beta1 lower than 0.9 starting point, do we just use config parameter: adam_beta1?

Thanks.

[jwm1969]

My speculation from looking at code is:

1. decouple is True by default
2. Beta1 is set in axolotl config like this:
   optimizer: adopt_adamw
   optimizer_kwargs:
   betas: [0.85, 0.999]

[bursteratom]

Hi @jwm1969 thank you for your question, and your guesses are completely correct! 'decoupled' is required to be set to true in order for ADOPT optimizer to work. This is already done internally within Axolotl codebase so the users won't need to worry about that.

And yes the beta values for adopt is share the same input as other adamw type optimizer. So just set adam_beta1 and adam_beta2 to whatever you desire in the config.

[jwm1969]

aha, thanks, so the syntax is:

adopt_beta1=0.85
adopt_beta2=0.999

not what i put above. Thanks.

[bursteratom]

Hi sorry I have mistyped. It's adam_beta1 and adam_beta2 i.e. same beta inputs for all adamw type optimizer