Skip to content

Commit

Permalink
More typos
Browse files Browse the repository at this point in the history
  • Loading branch information
Quentin-Anthony committed Sep 29, 2023
1 parent 5d52a1b commit 61f2554
Showing 1 changed file with 6 additions and 6 deletions.
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ GPT-NeoX leverages many of the same features and technologies as the popular Meg

**[8/10/2023]** We have experimental support for LLaMA 2 and Flash Attention v2 supported in our [math-lm](https://github.com/EleutherAI/math-lm) project that will be upstreamed later this month.

**[5/17/2023]** After fixing some miscellenous bugs we now fully support bf16.
**[5/17/2023]** After fixing some miscellaneous bugs we now fully support bf16.

**[4/11/2023]** We have upgraded our Flash Attention implementation to now support Alibi positional embeddings.

Expand Down Expand Up @@ -125,17 +125,17 @@ With your environment properly set up and the correct configuration files you ca

`python3 deepy.py train.py /path/to/configs/my_model.yml`

#### SLURM
#### Slurm

Using SLURM can be slightly more involved. Like with MPI, you must add the following to your config:
Using Slurm can be slightly more involved. Like with MPI, you must add the following to your config:

```json
{
"launcher": "slurm",
"deepspeed_slurm": true
}
```
If you do not have ssh access to the compute nodes in your SLURM cluster you need to add `{"no_ssh_check": true}`
If you do not have ssh access to the compute nodes in your Slurm cluster you need to add `{"no_ssh_check": true}`

#### (Advanced) Custom Launching

Expand Down Expand Up @@ -175,7 +175,7 @@ do
done
```

`$SLURM_JOBID` and `$SLURM_NODELIST` being environment variables SLURM will create for you. See the [sbatch documentation](https://slurm.schedmd.com/sbatch.html#SECTION_OUTPUT-ENVIRONMENT-VARIABLES) for a full list of available Slurm environment variables set at job creation time.
`$SLURM_JOBID` and `$SLURM_NODELIST` being environment variables Slurm will create for you. See the [sbatch documentation](https://slurm.schedmd.com/sbatch.html#SECTION_OUTPUT-ENVIRONMENT-VARIABLES) for a full list of available Slurm environment variables set at job creation time.

#### Job Launching

Expand Down Expand Up @@ -578,4 +578,4 @@ For full terms, see the `LICENSE` file. If you have any questions, comments, or

## Acknowledgements

We run our experiments on a Kubernetes cluster provided by [CoreWeave](https://coreweave.com/) and a SLURM cluster provided by [Stability AI](https://stability.ai). We are thankful to the DeepSpeed team for their advice and consultation.
We run our experiments on a Kubernetes cluster provided by [CoreWeave](https://coreweave.com/) and a Slurm cluster provided by [Stability AI](https://stability.ai). We are thankful to the DeepSpeed team for their advice and consultation.

0 comments on commit 61f2554

Please sign in to comment.