[Feature] Cautious optimizer, improve the stability of ADOPT optimizer, a new projector type random for GaLore optimizer

[kozistr]

Problem (Why?)

• implement Cautious optimizer
• improve the stability of ADOPT optimizer by normalizing and clipping the gradient
• support random protector type for GaLore optimizer

Solution (What/How?)

• implement Cautious optimizer
• improve the stability of ADOPT optimizer by normalizing and clipping the gradient
• support random protector type for GaLore optimizer

Other changes (bug fixes, small refactors)

N/A

Notes

N/A

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (131314b) to head (db82a58).
Report is 14 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #294   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           98        98           
  Lines         7400      7427   +27     
=========================================
+ Hits          7400      7427   +27     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.