Merge branch '812-megatron-seq-parallel' of https://github.com/Eleuth…

…erAI/gpt-neox into 812-megatron-seq-parallel

configs/neox_arguments.md

@@ -111,7 +111,7 @@ Logging Arguments
 
 - **git_hash**: str
 
-    Default = 455446c
+    Default = 3ccd3ba
 
     current git hash of repository
 
@@ -1056,6 +1056,16 @@ Parallelism Arguments
 
 
 
+- **sequence_parallel**: bool
+
+    Default = False
+
+    flag to determine whether Megatron-style (https://arxiv.org/abs/2205.05198) Sequence-Parallelism 
+    (sharding acts along seq. dim among TP group for LNs) will be used. Has no effect when model_parallel_size is 1.
+    **Set by user.**
+
+
+
 - **expert_interval**: int
 
     Default = 2