refactor(optimizer): use optimizer_cls_and_kwargs for custom optim

[NanoCode012]

Breaking change

• Deprecates alternate_optimizer. Move to use optimizer instead to consolidate.

Description

Refactors the optimizer handling to offload from create_optimizer into the Trainer instead. Only lora_plus requires overriding that function now.

• ao_adamw_4bit is available on transformers as adamw_torch_4bit.
• lion_pytorch is available on transformers as lion_32bit.
• Removed default handling of loraplus_lr_embedding to let it inherit from Model Input class instead
• Updated docs to include new upstream optimizers

TODO:

• Verify adamw_anyprecision is not affected
• Verify Galore is not affected
• Discuss whether to support other optimi optimizers https://optimi.benjaminwarner.dev/which_optimizer/
• Update transformers to commit that includes the linked PR
• Test custom optimizers
• Test upstream optimizers

Motivation and Context

This is due to upstream PR opening a new config huggingface/transformers#34358

How has this been tested?

Screenshots (if appropriate)

Types of changes

Social Handles (Optional)

Next steps

Once some time passes:

• Deprecate lion_pytorch and remove the dependency

[winglian]

@NanoCode012 now that 4.47.0 is out, we can re-open this.

[NanoCode012]

I noticed that the embedding scaled optimizer change was only applied for SFT trainer. Should it also be applied to DPOTrainer? @winglian