Skip to content

Commit

Permalink
[Feature & Bug] Refactor apis & Update examples & Fix re-import (#53)
Browse files Browse the repository at this point in the history
* refactor apis & update examples & fix re-import

* update process_hf_dataset

* add example readme

* fix conflict

* adapt new dataset

* passed examples and train.py

* Update examples/README.md

Co-authored-by: LZHgrla <[email protected]>

* Update examples/README.md

Co-authored-by: LZHgrla <[email protected]>

* Update examples/README.md

Co-authored-by: LZHgrla <[email protected]>

* Update examples/README.md

Co-authored-by: LZHgrla <[email protected]>

* del slurm_train.sh

* add 5 datasets apis

* Update xtuner/apis/model.py

Co-authored-by: LZHgrla <[email protected]>

* Update xtuner/apis/model.py

Co-authored-by: LZHgrla <[email protected]>

* Update xtuner/apis/model.py

Co-authored-by: LZHgrla <[email protected]>

---------

Co-authored-by: LZHgrla <[email protected]>
  • Loading branch information
humu789 and LZHgrla authored Aug 29, 2023
1 parent ff12607 commit 9f03ddf
Show file tree
Hide file tree
Showing 147 changed files with 911 additions and 1,524 deletions.
96 changes: 20 additions & 76 deletions examples/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# How to use XTuner in HuggingFace & DeepSpeed training pipelines
# How to use XTuner in HuggingFace training pipeline

## Quick run

Expand All @@ -11,86 +11,30 @@
2. run training scripts

```shell
# HuggingFace training pipeline
bash run_train_alpaca-qlora_with_xtuner_hf.sh
# DeepSpeed training pipeline
bash run_train_alpaca-qlora_with_xtuner_deepspeed.sh
```

3. (optional) whether to use `qlora` / `lora` or not

you can choose whether to use `qlora` / `lora` in training just by change the setting in scripts.

# qlora-training internlm-7b with alpaca dataset
python train_qlora_hf.py --model_name_or_path internlm/internlm-7b --dataset_name_or_path tatsu-lab/alpaca
```
# case 1. use qlora
--use_qlora True
--use_lora False

# case 2. use lora
--use_qlora False
--use_lora True

# case 3. neither
--use_qlora False
--use_lora False
```

## Training pipeline

If you want to use xtuner for efficient finetuning in your original training pipelines, you just need to change the implement of building `model` and `dataloader`, reserve other parts. Thus you can quickly fine-tune various models with various datasets by changing the relevant configs.
`--model_name_or_path`: specify the model name or path to train.

## How to build model with XTuner for QLoRA
`--dataset_name_or_path`: specify the dataset name or path to use.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from xtuner.models import SupervisedFinetune
## How to customize your experiment

model_name_or_path = 'internlm/internlm-7b'
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path,
trust_remote_code=True)
# if use lora, `quantization_config = None`
quantization_config = BitsAndBytesConfig(
load_in_4bit=True,
load_in_8bit=False,
llm_int8_threshold=6.0,
llm_int8_has_fp16_weight=False,
bnb_4bit_compute_dtype=torch.float16,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type='nf4')
# if neither use QLoRA or LoRA, `lora_config = None`
lora_config = LoraConfig(
r=64,
lora_alpha=16,
lora_dropout=0.1,
bias='none',
task_type='CAUSAL_LM')
# build base llm model
llm = AutoModelForCausalLM.from_pretrained(
model_name_or_path,
quantization_config=quantization_config,
trust_remote_code=True
)
# build model with XTuner
model = SupervisedFinetune(llm=llm, lora=lora_config, tokenizer=tokenizer)
XTuner APIs are compatible with the usage of HuggingFace's transformers.
If you want to customize your experiment, you just need to pass in your hyperparameters like HuggingFace.
```

## How to build train_dataloader

```python
from .data_utils import get_train_dataloader
dataset_cfg_path = '../configs/_base_/datasets/alpaca.py'
# need to pass in `tokenizer` previously prepared
train_dataloader = get_train_dataloader(dataset_cfg_path, tokenizer)
# (optional) you can get also dataset or collate_fn by train_dataloader
train_dataset = train_dataloader.dataset
data_collator = train_dataloader.collate_fn
# training example
python train_qlora_hf.py \
# custom training args
--model_name_or_path internlm/internlm-7b \
--dataset_name_or_path tatsu-lab/alpaca \
# HuggingFace's default training args
--do_train = True
--per_device_train_batch_size = 1
--learning_rate = 2e-5
--save_strategy = 'epoch'
--lr_scheduler_type = 'cosine'
--logging_steps = 1
```
14 changes: 0 additions & 14 deletions examples/data_utils.py

This file was deleted.

31 changes: 0 additions & 31 deletions examples/deepspeed_config.json

This file was deleted.

5 changes: 0 additions & 5 deletions examples/run_train_alpaca-qlora_with_mmchat_deepspeed.sh

This file was deleted.

22 changes: 0 additions & 22 deletions examples/run_train_alpaca-qlora_with_mmchat_hf.sh

This file was deleted.

145 changes: 0 additions & 145 deletions examples/train_alpaca-qlora_with_mmchat_deepspeed.py

This file was deleted.

Loading

0 comments on commit 9f03ddf

Please sign in to comment.