add main correctness plots to readme

karpathy · Sep 19, 2024 · 3756e8f · 3756e8f
1 parent 394547d
commit 3756e8f
Show file tree

Hide file tree

Showing 5 changed files with 23 additions and 1 deletion.
diff --git a/README.md b/README.md
@@ -1,7 +1,7 @@
 
 # nanoGPT-mup
 
-This repository is a fork of [nanoGPT](https://github.com/karpathy/nanoGPT) that provides a minimal implementation of the [maximal update parameterization](https://arxiv.org/abs/2203.03466) ([muP](https://github.com/microsoft/mup)) and acts as supplementary material for ["The Practitioner’s Guide to the Maximal Update Parameterization"](https://www.cerebras.ai/blog/the-practitioners-guide-to-the-maximal-update-parameterization). The `mup_examples` folder contains scripts to reproduce the plots in the blog post (see `mup_examples/README.md` for instructions to reproduce). 
+This repository is a fork of [nanoGPT](https://github.com/karpathy/nanoGPT) that provides a minimal implementation of the [maximal update parameterization](https://arxiv.org/abs/2203.03466) ([muP](https://github.com/microsoft/mup)) and acts as supplementary material for ["The Practitioner’s Guide to the Maximal Update Parameterization"](https://www.cerebras.ai/blog/the-practitioners-guide-to-the-maximal-update-parameterization). The [mup_examples](https://github.com/EleutherAI/nanoGPT-mup/tree/master/mup_examples) folder contains scripts to reproduce the plots in the blog post (see [mup_examples/README.md](https://github.com/EleutherAI/nanoGPT-mup/blob/master/mup_examples/README.md) for instructions to reproduce). 
 
 Each of the critical muP changes are marked with
 ```
@@ -12,6 +12,28 @@ Each of the critical muP changes are marked with
 to make everything easily searchable.
 
 
+## Implementation Validation
+
+We've verified the correctness of this implementation via coordinate checks and LR transfer.
+
+### Coordinate Checks
+
+Standard Parameterization:
+
+<img src="assets/coord_check_sp.png" alt="SP" width="60%">
+
+muTransfer:
+
+<img src="assets/coord_check_mup.png" alt="muP" width="60%">
+
+
+### Learning Rate muTransfer
+
+**Tiny Shakespeare**    |    **OpenWebText**
+:-------------------------:|:-------------------------:
+<img src="assets/mutransfer_lr_shakespeare_char.png" alt="mup-shakespeare" width="100%">     |  <img src="assets/mutransfer_lr_owt.png" alt="mup-owt" width="100%"> 
+
+
 If ["The Practitioner’s Guide to the Maximal Update Parameterization"](https://www.cerebras.ai/blog/the-practitioners-guide-to-the-maximal-update-parameterization) or this repository was useful to you, please cite:
 ```
 @misc{cerebras2024mupguide,

diff --git a/assets/coord_check_mup.png b/assets/coord_check_mup.png
diff --git a/assets/coord_check_sp.png b/assets/coord_check_sp.png
diff --git a/assets/mutransfer_lr_owt.png b/assets/mutransfer_lr_owt.png
diff --git a/assets/mutransfer_lr_shakespeare_char.png b/assets/mutransfer_lr_shakespeare_char.png