fix description and format error in docs (#106)

**Description** fix description and format error in docs
Azure · Oct 20, 2023 · ceadfea · ceadfea
1 parent 4784d99
commit ceadfea
Show file tree

Hide file tree

Showing 2 changed files with 5 additions and 2 deletions.
diff --git a/docs/user-tutorial/container-images.mdx b/docs/user-tutorial/container-images.mdx
@@ -25,8 +25,8 @@ You can use MS-AMP image by `ghcr.io/microsoft/msamp:${tag}`, available tags are
 
 | Tag               | Description                        |
 |-------------------|------------------------------------|
-| v0.2.0-cuda12.1   | MS-AMP v0.9.0 with CUDA 12.1       |
-| v0.2.0-cuda11.8   | MS-AMP v0.9.0 with CUDA 11.8     |
+| v0.2.0-cuda12.1   | MS-AMP v0.2.0 with CUDA 12.1       |
+| v0.2.0-cuda11.8   | MS-AMP v0.2.0 with CUDA 11.8       |
 
 </TabItem>
 </Tabs>
diff --git a/docs/user-tutorial/optimization-level.md b/docs/user-tutorial/optimization-level.md
@@ -2,6 +2,8 @@
 id: optimization-level
 ---
 
+# Optimization Level
+
 Currently MS-AMP supports three optimization levels: O1 and O2 and O3. The three levels gradually incorporate 8-bit collective communcation, optimizer and distributed parallel training in an incremental manner. Users can directly set O1/O2 using `msamp.initialize` and set O3 in config file when using DeepSpeed.
 
 - O1: We found that directly transitioning weight gradients from FP32 to FP8 in the Transformer Engine leads to a decrease in accuracy. However, this issue is resolved in O1 through the implementation of FP8 for weight gradients and AllReduce communication. This optimization also has the added benefits of saving GPU memory and reducing communication bandwidth.
@@ -11,6 +13,7 @@ Currently MS-AMP supports three optimization levels: O1 and O2 and O3. The three
 - O3: This optimization level is specifically designed for FP8 support in distributed parallel training for large scale models. These frequently-used strategies include data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism and ZeRO optimizer. ZeRO separates model weights into regular weights and master weights, with the former used for network forward/backward on each GPU, and the latter used for model updating in the optimizer. This separation allows us to use 8-bit data precision for regular weights and weight broadcasting, which reduces GPU memory and bandwidth usage even further.
 
 Here are details of different MS-AMP optimization levels:
+
 | Optimization Level  | Computation(GEMM) | Comm  | Weight | Master Weight  | Weight Gradient | Optimizer States |
 | ------------------- | -----------       | ----- | ------ | -------------  | --------------- | ---------------- |
 | FP16 AMP            | FP16              | FP32  | FP32   | N/A            | FP32            | FP32+FP32        |