Remove the NeoX implementation of GPT2Tokenizer (#1042)

* Try out just using the HF implementation Signed-off-by: Dashiell Stander <[email protected]> * Rely solely on HF tokenizer. Signed-off-by: Dashiell Stander <[email protected]> * Update NeoXArgs docs automatically --------- Signed-off-by: Dashiell Stander <[email protected]> Co-authored-by: github-actions <[email protected]>
EleutherAI · Sep 25, 2023 · 2ab05be · 2ab05be
1 parent e431ff5
commit 2ab05be
Show file tree

Hide file tree

Showing 3 changed files with 1 addition and 370 deletions.
diff --git a/configs/neox_arguments.md b/configs/neox_arguments.md
@@ -111,7 +111,7 @@ Logging Arguments
 
 - **git_hash**: str
 
-    Default = 1f832c1
+    Default = 3f8c63c
 
     current git hash of repository
 

diff --git a/megatron/tokenizer/gpt2_tokenization.py b/megatron/tokenizer/gpt2_tokenization.py
diff --git a/megatron/tokenizer/tokenizer.py b/megatron/tokenizer/tokenizer.py
@@ -25,7 +25,6 @@
 import numpy as np
 import sentencepiece as spm
 from typing import List, Union
-from .gpt2_tokenization import GPT2Tokenizer
 
 
 def build_tokenizer(args):