QWEN-2.5 DPO训练损失下降问题

[zifyloo]

Reminder

• I have read the README and searched the existing issues.

System Info

不同大小的数据集，如2k、6k，loss都在200step后下降缓慢，请问是哪里设置有问题吗

Reproduction

model

model_name_or_path: /home/ma-user/work/llama-factory/sft_v8_1019 # CodeQwen1.5-7B

method

stage: dpo
do_train: true
finetuning_type: lora
lora_target: all
pref_beta: 0.1
pref_ftx: 0.1
pref_loss: sigmoid

dataset

dataset: dpo_data
template: qwen
cutoff_len: 4096
max_samples: 30000
overwrite_cache: true
preprocessing_num_workers: 16

output

output_dir: /data/dpo/
logging_steps: 1
save_steps: 200
plot_loss: true
overwrite_output_dir: true

train

per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 5.0e-6
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
max_steps: 1200

eval

val_size: 0.1
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 200

Others

No response

[hiyouga]

已经收敛了