Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MiniLLM and Data selection #285

Merged
merged 8 commits into from
Nov 23, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update env
t1101675 committed Nov 23, 2024
commit 3fddecde3a3be8f4d5c145e77f9adc0735377034
22 changes: 11 additions & 11 deletions minillm/README.md
Original file line number Diff line number Diff line change
@@ -6,21 +6,21 @@

![Results](./figures/results.png)

See also:
+ [DPKD](https://github.com/microsoft/LMOps/tree/main/dpkd): A simple improvement of MiniLLM using DPO.
+ [MiniPLM](https://github.com/thu-coai/MiniPLM): Knowledge distillation for **pre-training** lanuage models.

## 1 Environment
```bash
pip3 install -e transformers/
pip3 install torch==2.0.1
pip3 install deepspeed==0.10.0
pip3 install torchvision==0.15.2
pip3 install nltk
pip3 install https://github.com/t1101675/transformers.git@minillm
pip3 install torch
pip3 install deepspeed
pip3 install numerize
pip3 install rouge-score
pip3 install torchtyping
pip3 install rich
pip3 install accelerate
pip3 install datasets
pip3 install sentencepiece
pip3 install protobuf==3.20.3
pip3 install peft
```
or
@@ -149,10 +149,10 @@ Multi-Node training is launched by `deepspeed`. We provide an example script in

## 6 Citation
```bibtex
@article{minillm,
title={Knowledge Distillation of Large Language Models},
@inproceedings{minillm,
title={MiniLLM: Knowledge Distillation of Large Language Models},
author={Gu, Yuxian and Dong, Li and Wei, Furu and Huang, Minlie},
journal={arXiv preprint arXiv:2306.08543},
year={2023}
booktitle={Proceedings of ICLR},
year={2024}
}
```
10 changes: 3 additions & 7 deletions minillm/install.sh
Original file line number Diff line number Diff line change
@@ -1,15 +1,11 @@
export NCCL_DEBUG=""
pip3 install -e transformers/
pip3 install torch==2.0.1
pip3 install deepspeed==0.10.0
pip3 install torchvision==0.15.2
pip3 install nltk
pip3 install https://github.com/t1101675/transformers.git@minillm
pip3 install torch
pip3 install deepspeed
pip3 install numerize
pip3 install rouge-score
pip3 install torchtyping
pip3 install rich
pip3 install accelerate
pip3 install datasets
pip3 install sentencepiece
pip3 install protobuf==3.20.3
pip3 install peft