Merge branch 'open-mmlab:main' into pipeline

docs/en/api/visualization.rst

@@ -36,3 +36,5 @@ visualization Backend
    WandbVisBackend
    ClearMLVisBackend
    NeptuneVisBackend
+   DVCLiveVisBackend
+   AimVisBackend

docs/en/common_usage/better_optimizers.md

@@ -90,3 +90,66 @@ runner = Runner(
 )
 runner.train()
 ```
+
+## bitsandbytes
+
+[bitsandbytes](https://github.com/TimDettmers/bitsandbytes) provides `AdamW8bit`, `Adam8bit`, `Adagrad8bit`, `PagedAdam8bit`, `PagedAdamW8bit`, `LAMB8bit`, `LARS8bit`, `RMSprop8bit`, `Lion8bit`, `PagedLion8bit` and `SGD8bit` optimziers。
+
+```{note}
+If you use the optimizer provided by bitsandbytes, you need to upgrade mmengine to `0.8.5`.
+```
+
+- Installation
+
+```bash
+pip install bitsandbytes
+```
+
+- Usage
+
+Take the `AdamW8bit` as an example.
+
+```python
+runner = Runner(
+    model=ResNet18(),
+    work_dir='./work_dir',
+    train_dataloader=train_dataloader_cfg,
+    # To view the input parameters for AdamW8bit, you can refer to
+    # https://github.com/TimDettmers/bitsandbytes/blob/main/bitsandbytes/optim/adamw.py
+    optim_wrapper=dict(optimizer=dict(type='AdamW8bit', lr=1e-4, weight_decay=1e-2)),
+    train_cfg=dict(by_epoch=True, max_epochs=3),
+)
+runner.train()
+```
+
+## transformers
+
+[transformers](https://github.com/huggingface/transformers) provides `Adafactor` optimzier。
+
+```{note}
+If you use the optimizer provided by transformers, you need to upgrade mmengine to `0.8.5`.
+```
+
+- Installation
+
+```bash
+pip install transformers
+```
+
+- Usage
+
+Take the `Adafactor` as an example.
+
+```python
+runner = Runner(
+    model=ResNet18(),
+    work_dir='./work_dir',
+    train_dataloader=train_dataloader_cfg,
+    # To view the input parameters for Adafactor, you can refer to
+    # https://github.com/huggingface/transformers/blob/v4.33.2/src/transformers/optimization.py#L492
+    optim_wrapper=dict(optimizer=dict(type='Adafactor', lr=1e-5,
+        weight_decay=1e-2, scale_parameter=False, relative_step=False)),
+    train_cfg=dict(by_epoch=True, max_epochs=3),
+)
+runner.train()
+```

docs/en/common_usage/visualize_training_log.md

@@ -1,6 +1,6 @@
 # Visualize Training Logs
 
-MMEngine integrates experiment management tools such as [TensorBoard](https://www.tensorflow.org/tensorboard), [Weights & Biases (WandB)](https://docs.wandb.ai/), [MLflow](https://mlflow.org/docs/latest/index.html), [ClearML](https://clear.ml/docs/latest/docs) and [Neptune](https://docs.neptune.ai/), making it easy to track and visualize metrics like loss and accuracy.
+MMEngine integrates experiment management tools such as [TensorBoard](https://www.tensorflow.org/tensorboard), [Weights & Biases (WandB)](https://docs.wandb.ai/), [MLflow](https://mlflow.org/docs/latest/index.html), [ClearML](https://clear.ml/docs/latest/docs), [Neptune](https://docs.neptune.ai/), [DVCLive](https://dvc.org/doc/dvclive) and [Aim](https://aimstack.readthedocs.io/en/latest/overview.html), making it easy to track and visualize metrics like loss and accuracy.
 
 Below, we'll show you how to configure an experiment management tool in just one line, based on the example from [15 minutes to get started with MMEngine](../get_started/15_minutes.md).
 
@@ -149,3 +149,88 @@ runner.train()
 ```
 
 More initialization configuration parameters are available at [neptune.init_run API](https://docs.neptune.ai/api/neptune/#init_run).
+
+## DVCLive
+
+Before using DVCLive, you need to install `dvclive` dependency library and refer to [iterative.ai](https://dvc.org/doc/start) for configuration. Common configurations are as follows:
+
+```bash
+pip install dvclive
+cd ${WORK_DIR}
+git init
+dvc init
+git commit -m "DVC init"
+```
+
+Configure the `Runner` in the initialization parameters of the Runner, and set `vis_backends` to [DVCLiveVisBackend](mmengine.visualization.DVCLiveVisBackend).
+
+```python
+runner = Runner(
+    model=MMResNet50(),
+    work_dir='./work_dir_dvc',
+    train_dataloader=train_dataloader,
+    optim_wrapper=dict(optimizer=dict(type=SGD, lr=0.001, momentum=0.9)),
+    train_cfg=dict(by_epoch=True, max_epochs=5, val_interval=1),
+    val_dataloader=val_dataloader,
+    val_cfg=dict(),
+    val_evaluator=dict(type=Accuracy),
+    visualizer=dict(type='Visualizer', vis_backends=[dict(type='DVCLiveVisBackend')]),
+)
+runner.train()
+```
+
+```{note}
+Recommend not to set `work_dir` as `work_dirs`. Or DVC will give a warning `WARNING:dvclive:Error in cache: bad DVC file name 'work_dirs\xxx.dvc' is git-ignored` if you run experiments in a OpenMMLab's repo.
+```
+
+Open the `report.html` file under `work_dir_dvc`, and you will see the visualization as shown in the following image.
+
+![image](https://github.com/open-mmlab/mmengine/assets/58739961/47d85520-9a4a-4143-a449-12ed7347cc63)
+
+You can also configure a VSCode extension of [DVC](https://marketplace.visualstudio.com/items?itemName=Iterative.dvc) to visualize the training process.
+
+More initialization configuration parameters are available at [DVCLive API Reference](https://dvc.org/doc/dvclive/live).
+
+## Aim
+
+Before using Aim, you need to install `aim` dependency library.
+
+```bash
+pip install aim
+```
+
+Configure the `Runner` in the initialization parameters of the Runner, and set `vis_backends` to [AimVisBackend](mmengine.visualization.AimVisBackend).
+
+```python
+runner = Runner(
+    model=MMResNet50(),
+    work_dir='./work_dir',
+    train_dataloader=train_dataloader,
+    optim_wrapper=dict(optimizer=dict(type=SGD, lr=0.001, momentum=0.9)),
+    train_cfg=dict(by_epoch=True, max_epochs=5, val_interval=1),
+    val_dataloader=val_dataloader,
+    val_cfg=dict(),
+    val_evaluator=dict(type=Accuracy),
+    visualizer=dict(type='Visualizer', vis_backends=[dict(type='AimVisBackend')]),
+)
+runner.train()
+```
+
+In the terminal, use the following command,
+
+```bash
+aim up
+```
+
+or in the Jupyter Notebook, use the following command,
+
+```bash
+%load_ext aim
+%aim up
+```
+
+to launch the Aim UI as shown below.
+
+![image](https://github.com/open-mmlab/mmengine/assets/58739961/2fc6cdd8-1de7-4125-a20a-c95c1a8bdb1b)
+
+Initialization configuration parameters are available at [Aim SDK Reference](https://aimstack.readthedocs.io/en/latest/refs/sdk.html#module-aim.sdk.run).

docs/zh_cn/api/visualization.rst

@@ -36,3 +36,5 @@ visualization Backend
    WandbVisBackend
    ClearMLVisBackend
    NeptuneVisBackend
+   DVCLiveVisBackend
+   AimVisBackend

docs/zh_cn/common_usage/better_optimizers.md

@@ -90,3 +90,34 @@ runner = Runner(
 )
 runner.train()
 ```
+
+## bitsandbytes
+
+[bitsandbytes](https://github.com/TimDettmers/bitsandbytes) 提供了 `AdamW8bit`、`Adam8bit`、`Adagrad8bit`、`PagedAdam8bit`、`PagedAdamW8bit`、`LAMB8bit`、 `LARS8bit`、`RMSprop8bit`、`Lion8bit`、`PagedLion8bit` 和 `SGD8bit` 优化器。
+
+```{note}
+如使用 D-Adaptation 提供的优化器，需将 mmengine 升级至 `0.8.5`。
+```
+
+- 安装
+
+```bash
+pip install bitsandbytes
+```
+
+- 使用
+
+以 `AdamW8bit` 为例。
+
+```python
+runner = Runner(
+    model=ResNet18(),
+    work_dir='./work_dir',
+    train_dataloader=train_dataloader_cfg,
+    # 如需查看 AdamW8bit 的输入参数，可查看
+    # https://github.com/TimDettmers/bitsandbytes/blob/main/bitsandbytes/optim/adamw.py
+    optim_wrapper=dict(optimizer=dict(type='AdamW8bit', lr=1e-4, weight_decay=1e-2)),
+    train_cfg=dict(by_epoch=True, max_epochs=3),
+)
+runner.train()
+```

docs/zh_cn/common_usage/visualize_training_log.md

@@ -1,8 +1,8 @@
 # 可视化训练日志
 
-MMEngine 集成了 [TensorBoard](https://www.tensorflow.org/tensorboard?hl=zh-cn)、[Weights & Biases (WandB)](https://docs.wandb.ai/)、[MLflow](https://mlflow.org/docs/latest/index.html) 、[ClearML](https://clear.ml/docs/latest/docs) 和 [Neptune](https://docs.neptune.ai/) 实验管理工具，你可以很方便地跟踪和可视化损失及准确率等指标。
+MMEngine 集成了 [TensorBoard](https://www.tensorflow.org/tensorboard?hl=zh-cn)、[Weights & Biases (WandB)](https://docs.wandb.ai/)、[MLflow](https://mlflow.org/docs/latest/index.html) 、[ClearML](https://clear.ml/docs/latest/docs)、[Neptune](https://docs.neptune.ai/)、[DVCLive](https://dvc.org/doc/dvclive) 和 [Aim](https://aimstack.readthedocs.io/en/latest/overview.html) 实验管理工具，你可以很方便地跟踪和可视化损失及准确率等指标。
 
-下面基于[15 分钟上手 MMENGINE](../get_started/15_minutes.md)中的例子介绍如何一行配置实验管理工具。
+下面基于 [15 分钟上手 MMENGINE](../get_started/15_minutes.md) 中的例子介绍如何一行配置实验管理工具。
 
 ## TensorBoard
 
@@ -149,3 +149,88 @@ runner.train()
 ```
 
 更多初始化配置参数可点击 [neptune.init_run API](https://docs.neptune.ai/api/neptune/#init_run) 查询。
+
+## DVCLive
+
+使用 DVCLive 前需先安装依赖库 `dvclive` 并参考 [iterative.ai](https://dvc.org/doc/start) 进行配置。常见的配置方式如下：
+
+```bash
+pip install dvclive
+cd ${WORK_DIR}
+git init
+dvc init
+git commit -m "DVC init"
+```
+
+设置 `Runner` 初始化参数中的 `visualizer`，并将 `vis_backends` 设置为 [DVCLiveVisBackend](mmengine.visualization.DVCLiveVisBackend)。
+
+```python
+runner = Runner(
+    model=MMResNet50(),
+    work_dir='./work_dir_dvc',
+    train_dataloader=train_dataloader,
+    optim_wrapper=dict(optimizer=dict(type=SGD, lr=0.001, momentum=0.9)),
+    train_cfg=dict(by_epoch=True, max_epochs=5, val_interval=1),
+    val_dataloader=val_dataloader,
+    val_cfg=dict(),
+    val_evaluator=dict(type=Accuracy),
+    visualizer=dict(type='Visualizer', vis_backends=[dict(type='DVCLiveVisBackend')]),
+)
+runner.train()
+```
+
+```{note}
+推荐将 `work_dir` 设置为 `work_dirs`。否则，你在 OpenMMLab 仓库中运行试验时，DVC 会给出警告 `WARNING:dvclive:Error in cache: bad DVC file name 'work_dirs\xxx.dvc' is git-ignored`。
+```
+
+打开 `work_dir_dvc` 下面的 `report.html` 文件，即可看到如下图的可视化效果。
+
+![image](https://github.com/open-mmlab/mmengine/assets/58739961/47d85520-9a4a-4143-a449-12ed7347cc63)
+
+你还可以安装 VSCode 扩展 [DVC](https://marketplace.visualstudio.com/items?itemName=Iterative.dvc) 进行可视化。
+
+更多初始化配置参数可点击 [DVCLive API Reference](https://dvc.org/doc/dvclive/live) 查询。
+
+## Aim
+
+使用 Aim 前需先安装依赖库 `aim`。
+
+```bash
+pip install aim
+```
+
+设置 `Runner` 初始化参数中的 `visualizer`，并将 `vis_backends` 设置为 [AimVisBackend](mmengine.visualization.AimVisBackend)。
+
+```python
+runner = Runner(
+    model=MMResNet50(),
+    work_dir='./work_dir',
+    train_dataloader=train_dataloader,
+    optim_wrapper=dict(optimizer=dict(type=SGD, lr=0.001, momentum=0.9)),
+    train_cfg=dict(by_epoch=True, max_epochs=5, val_interval=1),
+    val_dataloader=val_dataloader,
+    val_cfg=dict(),
+    val_evaluator=dict(type=Accuracy),
+    visualizer=dict(type='Visualizer', vis_backends=[dict(type='AimVisBackend')]),
+)
+runner.train()
+```
+
+在终端中输入
+
+```bash
+aim up
+```
+
+或者在 Jupyter Notebook 中输入
+
+```bash
+%load_ext aim
+%aim up
+```
+
+即可启动 Aim UI，界面如下图所示。
+
+![image](https://github.com/open-mmlab/mmengine/assets/58739961/2fc6cdd8-1de7-4125-a20a-c95c1a8bdb1b)
+
+初始化配置参数可点击 [Aim SDK Reference](https://aimstack.readthedocs.io/en/latest/refs/sdk.html#module-aim.sdk.run) 查询。

mmengine/config/config.py

@@ -17,6 +17,7 @@
 from pathlib import Path
 from typing import Any, Optional, Sequence, Tuple, Union
 
+import yapf
 from addict import Dict
 from rich.console import Console
 from rich.text import Text
@@ -1472,8 +1473,11 @@ def _format_dict(input_dict, outest_level=False):
                 blank_line_before_nested_class_or_def=True,
                 split_before_expression_after_opening_paren=True)
             try:
-                text, _ = FormatCode(
-                    text, style_config=yapf_style, verify=True)
+                if digit_version(yapf.__version__) >= digit_version('0.40.2'):
+                    text, _ = FormatCode(text, style_config=yapf_style)
+                else:
+                    text, _ = FormatCode(
+                        text, style_config=yapf_style, verify=True)
             except:  # noqa: E722
                 raise SyntaxError('Failed to format the config file, please '
                                   f'check the syntax of: \n{text}')

mmengine/hooks/logger_hook.py

@@ -46,7 +46,7 @@ class LoggerHook(Hook):
             checkpoints. If not specified, ``runner.work_dir`` will be used
             by default. If specified, the ``out_dir`` will be the concatenation
             of ``out_dir`` and the last level directory of ``runner.work_dir``.
-            For example, if the input ``our_dir`` is ``./tmp`` and
+            For example, if the input ``out_dir`` is ``./tmp`` and
             ``runner.work_dir`` is ``./work_dir/cur_exp``, then the log will be
             saved in ``./tmp/cur_exp``. Defaults to None.
         out_suffix (Tuple[str] or str): Those files in ``runner._log_dir``

mmengine/optim/optimizer/builder.py

@@ -129,6 +129,49 @@ def register_sophia_optimizers() -> List[str]:
 SOPHIA_OPTIMIZERS = register_sophia_optimizers()
 
 
+def register_bitsandbytes_optimizers() -> List[str]:
+    """Register optimizers in ``bitsandbytes`` to the ``OPTIMIZERS`` registry.
+
+    Returns:
+        List[str]: A list of registered optimizers' name.
+    """
+    dadaptation_optimizers = []
+    try:
+        import bitsandbytes as bnb
+    except ImportError:
+        pass
+    else:
+        for module_name in [
+                'AdamW8bit', 'Adam8bit', 'Adagrad8bit', 'PagedAdam8bit',
+                'PagedAdamW8bit', 'LAMB8bit', 'LARS8bit', 'RMSprop8bit',
+                'Lion8bit', 'PagedLion8bit', 'SGD8bit'
+        ]:
+            _optim = getattr(bnb.optim, module_name)
+            if inspect.isclass(_optim) and issubclass(_optim,
+                                                      torch.optim.Optimizer):
+                OPTIMIZERS.register_module(module=_optim)
+                dadaptation_optimizers.append(module_name)
+    return dadaptation_optimizers
+
+
+BITSANDBYTES_OPTIMIZERS = register_bitsandbytes_optimizers()
+
+
+def register_transformers_optimizers():
+    transformer_optimizers = []
+    try:
+        from transformers import Adafactor
+    except ImportError:
+        pass
+    else:
+        OPTIMIZERS.register_module(name='Adafactor', module=Adafactor)
+        transformer_optimizers.append('Adafactor')
+    return transformer_optimizers
+
+
+TRANSFORMERS_OPTIMIZERS = register_transformers_optimizers()
+
+
 def build_optim_wrapper(model: nn.Module,
                         cfg: Union[dict, Config, ConfigDict]) -> OptimWrapper:
     """Build function of OptimWrapper.