Skip to content

Latest commit

 

History

History
389 lines (349 loc) · 17.5 KB

changelog.md

File metadata and controls

389 lines (349 loc) · 17.5 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

[1.5.0] - 2022-02-15

This small release improves ResNet support of some second-order extensions and adds some more custom layers.

Thanks to @jabader97 for contributing to KFAC.

Added/New

  • KFAC for ResNets modules (commit)
  • Custom modules for
  • {Batch}DiagGGN{Exact,MC}, {Batch}DiagHessian support for
    • Custom slicing module (commit)
    • Custom padding module (commit)
  • SqrtGGN{Exact,MC} support for
    • BatchNormNd (eval mode) (commit)
    • ResNet modules (ScaleModule, SumModule, Identity) (commit)
    • Custom slicing module (commit)
    • Custom padding module (commit)

Fixed/Removed

Internal

  • Automate PyPI release through GitHub action (commit)
  • Add KFAC tests for linear layers to test suite (commit)
  • Update formatting to black==22.1.0 (commit)
  • Add BackPACK logo to RTD (commit)

1.4.0 - 2021-11-12

This release ships many new features. Some rely on recent PyTorch functionality. We now require torch>=1.9.0.

Highlights:

  • ResNets & RNNs: Thanks to @schaefertim for bringing basic support for RNNs (#16, tutorial) and ResNets (#14, tutorial)
  • SqrtGGN{Exact,MC} extension: Symmetric factorization of the generalized Gauss-Newton/Fisher (see arXiv:2106.02624)
  • Sub-sampling: Allows for restricting BackPACK extensions to a sub-set of samples in the mini-batch (#12, tutorial)

Added/New

  • Converter functionality for basic support of ResNets and RNNs [PR1, PR2, PR3]
  • New extensions:
    • SqrtGGNExact: Symmetric factorization of the exact GGN/Fisher [PR]
    • SqrtGGNMC: Symmetric factorization of the MC-approximated GGN/Fisher [PR]
  • Module support:
    • Linear: Support additional (more than 2) input dimensions [PR1, PR2]
    • BatchNormNd: Distinguish evaluation and training mode, support first-order extensions and DiagGGN{Exact,MC} [#160, PR1, PR2]
    • AdaptiveAvgPoolND: Support first-order extensions and DiagGGN{Exact,MC} [PR]
    • RNN: Support first-order extensions and DiagGGN{MC,Exact} [PR1 PR2 PR3]
    • LSTM: Support first-order extensions and DiagGGN{MC,Exact} [PR]
    • CrossEntropyLoss: Support additional (more than 2) input dimensions. [PR]
    • Embedding: Support first-order extensions and DiagGGN{MC,Exact} [PR]
  • Mini-batch sub-sampling
    • BatchGrad [PR1, PR2]
    • SqrtGGN{Exact,MC} [PR]
  • retain_graph option for backpack context [PR]
  • Assume batch axis always first [PR]

Fixed/Removed

  • Deprecate python3.6, require at least python3.7 [PR]

Internal

  • Use full_backward_hook for torch>=1.9.0 [PR]
  • Core
    • Implement derivatives for LSTM [PR]
    • Implement derivatives for AdaptiveAvgPoolNd [PR]
    • Sub-sampling
      • weight_jac_t_mat_prod [PR]
      • bias_jac_t_mat_prod [PR]
      • *_jac_t_mat_prod of RNN and LSTM parameters [PR]
      • jac_t_mat_prod [PR]
      • Hessian square root decomposition (exact and MC) [PR]
    • Refactor: Share code for *_jac_t_mat_prod [PR]
  • Extensions
    • Refactor BatchL2Grad, introducing a base class [PR]
    • Automate parameter functions for BatchGrad and Grad [PR]
    • Introduce interface to check module hyperparameters [PR]
  • Tests
    • Check if module Hessian is zero [PR]
    • Reduce run time [PR]

1.3.0 - 2021-06-16

Thanks to @sbharadwajj and @schaefertim for co-authoring many PRs shipped in this release.

Added

  • New extensions
    • BatchDiagGGN{Exact,MC}: Per sample diagonal of the GGN/Fisher, exact or with a Monte-Carlo approximation [PR1, PR2, PR3, example]
    • BatchDiagHessian: Per sample diagonal of the Hessian [PR1, PR2, example]
  • Support for more layers ([PR, overview])
    • DiagGGN{Exact,MC} extensions
      • Conv{1,3}d, ConvTranspose{1,2,3}d, LeakyReLU, LogSigmoid, ELU, SELU [PR]
      • MaxPool{1,3}d [PR]
      • AvgPool{1,3}d [PR]
    • DiagHessian extension
      • Conv{1,3}d, ConvTranspose{1,2,3}d, LeakyReLU, LogSigmoid [PR]
      • MaxPool{1,3}d [PR]
      • AvgPool{1,3}d [PR]
      • ELU, SELU [PR]
    • group argument of (transpose) convolutions
      • Full support for first-order diagonal curvature extensions [PR1, PR2, PR3, PR4]
      • No support (yet) for KFAC, KFLR and KFRA extensions [PR]
  • Extension hook which allows to run code right after a BackPACK extension [PR, example]
  • Context to disable BackPACK [PR]
  • Tutorial how to extend custom modules [PR, example]
  • (Experimental) Alternative convolution weight Jacobian with option to save memory [PR, example]

Fixed/Removed

  • Remove hooks that save input/output shapes. This probably resolves #97 [PR]
  • Remove DiagGGN from API (use DiagGGNExact instead). It was indented as abstract parent class for DiagGGNExact and DiagGGNMC [PR]

Internal

  • CI
    • Move tests from Travis to GitHub actions [PR, small fix]
    • Test DiagHessian with new test suite [PR]
    • Test DiagGGN with new test suite, introduce 'light' and 'full' tests [PR1, PR2]
    • Fix isort [PR]
    • Add partial docstring checks [PR]
    • Add docstrings to contribution guide lines [commit]
    • Auto-format and lint examples [PR]
  • Refactoring
    • Share code between Conv{Transpose}{1,2,3}d in BatchL2Grad [PR]
    • Use eingroup package, remove custom eingroup utility [PR]
  • Core
    • Implement derivatives for MaxPool{1,3}d [PR]
    • Implement derivatives for AvgPool{1,3}d [PR]
    • Support for groups in (transpose) convolutions [PR1, PR2, PR3, PR4]

1.2.0 - 2020-10-26

Thanks to @sbharadwajj for co-authoring many PRs shipped in this release.

Added

  • Deprecated python3.5, tested compatibility with PyTorch 1.6.0 [PR]
  • Support first-order extensions for Conv1d, Conv3d, ConvTranspose1d, ConvTranspose2d, ConvTranspose3d
    • extensions.BatchGrad [PR]
    • extensions.BatchL2Grad [PR]
    • extensions.SumGradSquared and extensions.Variance [PR]
    • Raise exceptions for unsupported exotic hyperparameters [PR1, PR2]
  • New example: Backpropagating through BackPACK quantities [commit]
  • New extensions in API: Block-diagonal curvature products
    • Exposed via extensions.HMP, extensions.GGNMP, extensions.PCHMP [PR]
    • Examples: Hutchinson trace estimation [PR] and Hessian-free optimization with CG [PR]

Fixed

  • Add missing zero_grad in the diagonal GGN second-order optimization example [PR]

Internal

  • Increased test coverage
    • New test suite for backpack.extensions [PR]
    • New test suite for backpack.core [PR]
  • Implemented derivatives of the following operations in backpack.core
    • More activation functions [PR]
    • Conv1d, Conv3d [PR]
    • ConvTranspose1d, ConvTranspose2d, ConvTranspose3d [PR]
  • Refactor firstorder extensions to share more code [PR1, PR2]
  • Removed detachs to support differentiating through quantities [PR]

1.1.1 - 2020-04-29

Added

  • Improved documentation, moved to ReadTheDocs [PR1, PR2, PR3]
  • Tested compatibility with PyTorch 1.5.0.
  • Support 2nd-order backprop for vectors in MSELoss [PR]
  • Sanity checks to raise warnings if the following are used. inplace modification [PR], unsupported loss parameters [PR], custom losses in 2nd-order backpropagation [PR]

Fixed

  • Removed opt_einsum dependency [PR]
  • Missing implementations and wrong backpropagation of KFRA for Conv2d, MaxPool2d, and AvgPool2d [PR]
  • Remove try_view and use reshape to use PyTorch 1.4.0 improvements [PR]

Internal

  • Docstring style [PR]

1.1.0 - 2020-02-11

Added

  • Support MC sampling [Issue, PR]
  • Utilities to handle Kronecker factors [PR]
  • Examples [PR]

Fixed

  • Fixed documentation issue in Batch l2 [PR]
  • Added support for stride parameter in Conv2d [Issue, PR]
  • Pytorch 1.3.0 compatibility [PR, PR]

Internal

  • Added continuous integration [PR], test coverage [PR], style guide enforcement [PR]
  • Changed internal shape conventions of backpropagated quantities for performance improvements [PR]

1.0.1 - 2019-09-05

Fixed

  • Fixed PyPI installaton

1.0.0 - 2019-10-03

Initial release