pytorch-optimizer v3.3.2

Change Log

Feature

• Implement SGDSaI optimizer. (#315, #316)
  • No More Adam: Learning Rate Scaling at Initialization is All You Need

Bug

• Clone exp_avg before calling apply_cautious not to mask exp_avg. (#316)

pytorch-optimizer v3.3.1

Change Log

Feature

• Support Cautious variant to AdaShift optimizer. (#310)
• Save the state of the Lookahead optimizer too. (#310)
• Implement APOLLO optimizer. (#311, #312)
  • SGD-like Memory, AdamW-level Performance
• Rename the Apollo (An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization) optimizer name to ApolloDQN not to overlap with the new optimizer name APOLLO. (#312)
• Implement MARS optimizer. (#313, #314)
  • Unleashing the Power of Variance Reduction for Training Large Models
• Support Cautious variant to MARS optimizer. (#314)

Bug

• Fix bias_correction in AdamG optimizer. (#305, #308)
• Fix a potential bug when loading the state for Lookahead optimizer. (#306, #310)

Docs

• Add more visualizations. (#310, #314)

Contributions

thanks to @Vectorrent

pytorch-optimizer v3.3.0

Change Log

Feature

• Support PaLM variant for ScheduleFreeAdamW optimizer. (#286, #288)
  • you can use this feature by setting use_palm to True.
• Implement ADOPT optimizer. (#289, #290)
  • Modified Adam Can Converge with Any β2 with the Optimal Rate
• Implement FTRL optimizer. (#291)
  • Follow The Regularized Leader
• Implement Cautious optimizer feature. (#294)
  • Improving Training with One Line of Code
  • you can use it by setting cautious=True for Lion, AdaFactor and AdEMAMix optimizers.
• Improve the stability of ADOPT optimizer. (#294)
  • Note
• Support a new projection type random for GaLoreProjector. (#294)
• Implement DeMo optimizer. (#300, #301)
  • Decoupled Momentum Optimization
• Implement Muon optimizer. (#302)
  • MomentUm Orthogonalized by Newton-schulz
• Implement ScheduleFreeRAdam optimizer. (#304)
• Implement LaProp optimizer. (#304)
  • Separating Momentum and Adaptivity in Adam
• Support Cautious variant to LaProp, AdamP, Adopt optimizers. (#304).

Refactor

• Big refactoring, removing direct import from pytorch_optimizer.*.
  • I removed some methods not to directly import from it from pytorch_optimzier.* because they're probably not used frequently and actually not an optimizer rather utils only used for specific optimizers.
  • pytorch_optimizer.[Shampoo stuff] -> pytorch_optimizer.optimizers.shampoo_utils.[Shampoo stuff].
    • shampoo_utils like Graft, BlockPartitioner, PreConditioner, etc. You can check the details here.
  • pytorch_optimizer.GaLoreProjector -> pytorch_optimizer.optimizers.galore.GaLoreProjector.
  • pytorch_optimizer.gradfilter_ema -> pytorch_optimizer.optimizers.grokfast.gradfilter_ema.
  • pytorch_optimizer.gradfilter_ma -> pytorch_optimizer.optimizers.grokfast.gradfilter_ma.
  • pytorch_optimizer.l2_projection -> pytorch_optimizer.optimizers.alig.l2_projection.
  • pytorch_optimizer.flatten_grad -> pytorch_optimizer.optimizers.pcgrad.flatten_grad.
  • pytorch_optimizer.un_flatten_grad -> pytorch_optimizer.optimizers.pcgrad.un_flatten_grad.
  • pytorch_optimizer.reduce_max_except_dim -> pytorch_optimizer.optimizers.sm3.reduce_max_except_dim.
  • pytorch_optimizer.neuron_norm -> pytorch_optimizer.optimizers.nero.neuron_norm.
  • pytorch_optimizer.neuron_mean -> pytorch_optimizer.optimizers.nero.neuron_mean.

Docs

• Add more visualizations. (#297)

Bug

• Add optimizer parameter to PolyScheduler constructor. (#295)

Contributions

thanks to @tanganke

pytorch-optimizer v3.2.0

Change Log

Feature

• Implement SOAP optimizer. (#275)
  • SOAP: Improving and Stabilizing Shampoo using Adam
• Support AdEMAMix variants. (#276)
  • bnb_ademamix8bit, bnb_ademamix32bit, bnb_paged_ademamix8bit, bnb_paged_ademamix32bit
• Support 8/4bit, fp8 optimizers. (#208, #281)
  • torchao_adamw8bit, torchao_adamw4bit, torchao_adamwfp8.
• Support a module-name-level (e.g. LayerNorm) weight decay exclusion for get_optimizer_parameters. (#282, #283)
• Implement CPUOffloadOptimizer, which offloads optimizer to CPU for single-GPU training. (#284)
• Support a regex-based filter for searching names of optimizers, lr schedulers, and loss functions.

Bug

• Fix should_grokfast condition when initialization. (#279, #280)

Contributions

thanks to @Vectorrent

pytorch-optimizer v3.1.2

Change Log

Feature

• Implement AdEMAMix optimizer. (#272)
  • THE ADEMAMIX OPTIMIZER: BETTER, FASTER, OLDER

Bug

• Add **kwargs to the parameters for dummy placeholder. (#270, #271)

pytorch-optimizer v3.1.1

Change Log

Feature

• Implement TRAC optimizer. (#263)
  • Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learning
• Support AdamW optimizer via create_optimizer(). (#263)
• Implement AdamG optimizer. (#264, #265)
  • Towards Stability of Parameter-free Optimization

Bug

• Handle the optimizers that only take the model instead of the parameters in create_optimizer(). (#263)
• Move the variable to the same device with the parameter. (#266, #267)

pytorch-optimizer v3.1.0

Change Log

Feature

• Implement AdaLomo optimizer. (#258)
  • Low-memory Optimization with Adaptive Learning Rate
• Support Q-GaLore optimizer. (#258)
  • Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.
  • you can use by optimizer = load_optimizer('q_galore_adamw8bit')
• Support more bnb optimizers. (#258)
  • bnb_paged_adam8bit, bnb_paged_adamw8bit, bnb_*_*32bit.
• Improve power_iteration() speed up to 40%. (#259)
• Improve reg_noise() (E-MCMC) speed up to 120%. (#260)
• Support disable_lr_scheduler parameter for Ranger21 optimizer to disable built-in learning rate scheduler. (#261)

Refactor

• Refactor AdamMini optimizer. (#258)
• Deprecate optional dependency, bitsandbytes. (#258)
• Move get_rms, approximate_sq_grad functions to BaseOptimizer for reusability. (#258)
• Refactor shampoo_utils.py. (#259)
• Add debias, debias_adam methods in BaseOptimizer. (#261)
• Refactor to use BaseOptimizer only, not inherit multiple classes. (#261)

Bug

• Fix several bugs in AdamMini optimizer. (#257)

Contributions

thanks to @sdbds

pytorch-optimizer v3.0.2

Change Log

Feature

• Implement WSD LR Scheduler. (#247, #248)
  • Warmup-Stable-Decay LR Scheduler
• Add more Pytorch built-in lr schedulers. (#248)
• Implement Kate optimizer. (#249, #251)
  • Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad
• Implement StableAdamW optimizer. (#250, #252)
  • Stable and low-precision training for large-scale vision-language models
• Implement AdamMini optimizer. (#246, #253)
  • Use Fewer Learning Rates To Gain More

Refactor

• Refactor Chebyschev lr scheduler modules. (#248)
  • Rename get_chebyshev_lr to get_chebyshev_lr_lambda.
  • Rename get_chebyshev_schedule to get_chebyshev_perm_steps.
  • Call get_chebyshev_schedule function to get LamdbaLR scheduler object.
• Refactor with ScheduleType. (#248)

pytorch-optimizer v3.0.1

Change Log

Feature

• Implement FAdam optimizer. (#241, #242)
  • Adam is a natural gradient optimizer using diagonal empirical Fisher information
• Tweak AdaFactor optimizer. (#236, #243)
  • support not-using-first-momentum when beta1 is not given
  • default dtype for first momentum to bfloat16
  • clip second momentum to 0.999
• Implement GrokFast optimizer. (#244, #245)
  • Accelerated Grokking by Amplifying Slow Gradients

Bug

• Wrong typing of reg_noise. (#239, #240)
• Lookahead`s param_groups attribute is not loaded from checkpoint. (#237, #238)

Contributions

thanks to @michaldyczko

pytorch-optimizer v3.0.0

Change Log

The major version is updated! (v2.12.0 -> v3.0.0) (#164)

Many optimizers, learning rate schedulers, and objective functions are in pytorch-optimizer.
Currently, pytorch-optimizer supports 67 optimizers (+ bitsandbytes), 11 lr schedulers, and 13 loss functions, and reached about 4 ~ 50K downloads / month (peak is 75K downloads / month)!

The reason for updating the major version from v2 to v3 is that I think it's a good time to ship the recent implementations (the last update was about 7 months ago) and plan to pivot to new concepts like training utilities while maintaining the original features (e.g. optimizers).
Also, rich test cases, benchmarks, and examples are on the list!

Finally, thanks for using the pytorch-optimizer, and feel free to make any requests :)

Feature

• Implement REX lr scheduler. (#217, #222)
  • Revisiting Budgeted Training with an Improved Schedule
• Implement Aida optimizer. (#220, #221)
  • A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range
• Implement WSAM optimizer. (#213, #216)
  • Sharpness-Aware Minimization Revisited: Weighted Sharpness as a Regularization Term
• Implement GaLore optimizer. (#224, #228)
  • Memory-Efficient LLM Training by Gradient Low-Rank Projection
• Implement Adalite optimizer. (#225, #229)
• Implement bSAM optimizer. (#212, #233)
  • SAM as an Optimal Relaxation of Bayes
• Implement Schedule-Free optimizer. (#230, #233)
  • Schedule-Free optimizers
• Implement EMCMC. (#231, #233)
  • Entropy-MCMC: Sampling from flat basins with ease

Fix

• Fix SRMM to allow operation beyond memory_length. (#227)

Dependency

• Drop Python 3.7 support officially. (#221)
  • Please check the README.
• Update bitsandbytes to 0.43.0. (#228)

Docs

• Add missing parameters in Ranger21 optimizer document. (#214, #215)
• Fix WSAM optimizer paper link. (#219)

Contributions

thanks to @sdbds, @i404788

Diff

• from the previous major version : 2.0.0...3.0.0
• from the previous version: 2.12.0...3.0.0