run_rm.sh双卡运行全量参数，报RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons:

[XiaozhuLove]

单卡不报错，双卡全量参数运行报如下错误：RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the forward function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple checkpoint functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations.
[rank0]: Parameter at index 286 with name model.layers.23.mlp.down_proj.weight has been marked as ready twice. This means that multiple autograd engine hooks have fired for this particular parameter during this iteration.
run_rm.h如下：
CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node 2 reward_modeling.py
--model_type auto
--model_name_or_path /home/xxxxxx/model/Qwen2-0.5B
--train_file_dir ./data/reward
--validation_file_dir ./data/reward
--per_device_train_batch_size 8
--per_device_eval_batch_size 4
--do_train
--use_peft False
--seed 42
--max_train_samples 1000
--max_eval_samples 10
--num_train_epochs 2
--learning_rate 2e-5
--warmup_ratio 0.05
--weight_decay 0.001
--logging_strategy steps
--logging_steps 10
--eval_steps 50
--evaluation_strategy steps
--save_steps 500
--save_strategy steps
--save_total_limit 3
--max_source_length 1024
--max_target_length 256
--output_dir outputs-rm-qwen-v1
--overwrite_output_dir
--ddp_timeout 30000
--logging_first_step True
--target_modules all
--lora_rank 8
--lora_alpha 16
--lora_dropout 0.05
--torch_dtype bfloat16
--device_map auto
--report_to tensorboard
--ddp_find_unused_parameters False
--remove_unused_columns False
--gradient_checkpointing True
--template_name qwen
谢谢！

[shibing624]

reward model 暂没多卡测试，先单卡跑。