Re-organize the folder

Co-authored-by: Stella Biderman <[email protected]> Signed-off-by: Dashiell Stander <[email protected]>
EleutherAI · Oct 2, 2023 · 94cc945 · 94cc945
1 parent 7a8569f
commit 94cc945
Show file tree

Hide file tree

Showing 20 changed files with 286 additions and 193 deletions.
diff --git a/prepare_data.py b/prepare_data.py
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from tools.corpora import prepare_dataset, DATA_DOWNLOADERS
+from tools.datasets.corpora import prepare_dataset, DATA_DOWNLOADERS
 import argparse
 
 TOKENIZER_CHOICES = [

diff --git a/tools/README.md b/tools/README.md
@@ -0,0 +1,15 @@
+# GPT-NeoX Auxiliary Tools
+
+This directory contains a number of auxiliary tools that are useful for working with GPT-NeoX but not part of the main training code.
+
+## Bash
+
+This directory contains some simple, frequently used bash commands to make working on multiple machines easier.
+
+## Checkpoints
+
+This directory contains tools for manipulating and converting checkpoints including changing the parallelism settings of a pretrained model, converting between GPT-NeoX and the transformers library, and updating checkpoints trained with Version 1.x of this library to be compatible with Version 2.x.
+
+## Datasets
+
+This directory contains tools for downloading and preprocessing datasets to the format expected by the GPT-NeoX library.
diff --git a/tools/kill.sh → tools/bash/kill.sh b/tools/kill.sh → tools/bash/kill.sh
diff --git a/tools/killall.sh → tools/bash/killall.sh b/tools/killall.sh → tools/bash/killall.sh
diff --git a/tools/sync.sh → tools/bash/sync.sh b/tools/sync.sh → tools/bash/sync.sh
diff --git a/tools/sync_cmd.sh → tools/bash/sync_cmd.sh b/tools/sync_cmd.sh → tools/bash/sync_cmd.sh
diff --git a/tools/syncdir.sh → tools/bash/syncdir.sh b/tools/syncdir.sh → tools/bash/syncdir.sh