pytorch-optimizer v2.3.0

Change Log

Feature

• re-implement Shampoo Optimizer (#97, related to #93)
  • layer-wise grafting (none, adagrad, sgd)
  • block partitioner
  • preconditioner
• remove casting to fp16 or bf16 inside of the step() not to lose consistency with the other optimizers. #96
• change some ops to in-place operations to speed up. #96

Fix

• fix exp_avg_var when amsgrad is True. #96

Refactor

• change linter from Pylint to Ruff, #97

pytorch-optimizer v2.2.1

Change Log

Feature

• Support max_grad_norm (Adan optimizer)
• Support gradient averaging (Lamb optimizer)
• Support dampening, nesterov parameters (Lars optimizer)

Refactor

• move step parameter from state to group. (to reduce computation cost & memory)
• load betas by group, not a parameter.
• change to in-place operations.

Fix

• fix when momentum is 0 (Lars optimizer)

pytorch-optimizer v2.2.0

Change Log

• Implement GSAM (Surrogate Gap Guided Sharpness-Aware Minimization) optimizer, ICLR 22

pytorch-optimizer v2.1.1

Change Log

#90

Feature

• Support gradient centralization for Adai optimizer
• Support AdamD debias for AdaPNM optimizer
• Register custom exceptions (e.g. NoSparseGradientError, NoClosureError, ...)

Documentation

• Add API documentation

Bug

• Fix SAM optimizer

pytorch-optimizer v2.1.0

Change Log

• Implement Adai optimizer, #89
• Speed up the testing, #88
• Upgrade to Python 3.11 (only for CI pipeline), #86

pytorch-optimizer v2.0.1

Change Log

• [Fix] update hubconf.py, #81
• [Fix] python 3.7 for Colab environment, #83 (#82)

pytorch-optimizer v2.0.0

Chage Log

• Refactor the package depth
  • 4 depths
    • pytorch_optimizer.lr_scheduler : lr schedulers
    • pytorch_optimizer.optimizer : optimizers
    • pytorch_optimizer.base : base utils
    • pytorch_optimizer.experimental : any experimental features
  • pytorch_optimizer.adamp -> pytorch_optimizer.optimizer.adamp
  • Still from pytorch_optimizer import AdamP is possible
• Implement lr schedulers
  • CosineAnealingWarmupRestarts
• Implement (experimental) lr schedulers
  • DeBERTaV3-large layer-wise lr scheduler

Other changes (bug fixes, small refactors)

• Fix AGC (to returning the parameter)
• Make a room for experimental features (at pytorch_optimizer.experimental)
• base types

pytorch-optimizer v1.3.2

Change Log

• torch.hub usage in docs, #76
• Adan optimizer, #77
  • fix: forgot to divide into beta_correction
  • feat: support weight_decouple

pytorch-optimizer v1.3.1

Change Log

• raw directive in RST format cannot be used due to the security issue e.g. code injection. #75

pytorch-optimizer v1.3.0

Change Log

• Support torch.hub.load to load the pytorch_optimizer, #73

Contributions

thanks to

@Bing-su