Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add LLaMAv2-70B text generation #441

Merged
merged 3 commits into from
Oct 3, 2023
Merged

Add LLaMAv2-70B text generation #441

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
be8828f
Add LLaMAv2-70B text generation
dvarshney-habana Oct 3, 2023
56df635
address review comments
dvarshney-habana Oct 3, 2023
ce4a91d
removed new line at end of command
dvarshney-habana Oct 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions examples/text-generation/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,21 @@ python ../gaudi_spawn.py --use_deepspeed --world_size 8 run_generation.py \
--max_new_tokens 100
```

You can also run Llama2-70B on Gaudi2 with all optimizations enabled using the following command:
```bash
python ../gaudi_spawn.py --use_deepspeed --world_size 8 run_generation.py \
--model_name_or_path meta-llama/Llama-2-70b-hf \
--max_new_tokens 4096 \
--bf16 \
--use_hpu_graphs \
--use_kv_cache \
--batch_size 52 \
--attn_softmax_bf16 \
--limit_hpu_graphs \
--reuse_cache \
--trim_logits
```

> To be able to run gated models like [StarCoder](https://huggingface.co/bigcode/starcoder), you should:
> - have a HF account
> - agree to the terms of use of the model in its model card on the HF Hub
Expand Down
Loading