Adai Optimizer ICML2022 Oral: adaptive momentum hyperparameters instead of adaptive learning rate

[zeke-xie]

Hi,

Our ICML2022 Oral paper proposed a novel adaptive optimization method named Adaptive Inertia (Adai), which uses parameter-wise inertia (the momentum hyperparameter as a vector) to accelerate saddle-point escaping and provably select flat minima as well as SGD.

Adai combines the advantages of Adam and SGD on saddle-point escaping and minima selection, respectively.

The official implementation can be found at https://github.com/zeke-xie/adaptive-inertia-adai
This may be helpful for your project.