How is stage 1 used in the next stage?

[raihan0824]

Hello, I am relatively new to this field, so I’m still confused a little about how the pipeline is structured. Stage 1 is supervised fine-tuning, the output is an sft model. My question is simple: How and where does the sft model used in the next stage?

Also, is there any interface/UI to generate the rlhf dataset? Because in my case, I want the model to be specific to my needs.

Also, what are the hardware requirements for training, and how long does it takes to complete the training?

Thank you!

[binmakeswell]

Hi @raihan0824 For 3 stages, you can refer to https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat#how-to-use
For dataset, you can refer to https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat#supervised-datasets-collection
You can reproduce it in a few hours using a single server.
We are putting together an example for as low resource (e.g. a consumer GPU) as possible (features are actually supported) and expect to make it available next week.

[raihan0824]

I'm just confused, after stage 1 (training sft model), I do not see where the sft model is used in stage 2. Because in stage 2, the model that is used is the llama model, not the sft model that has been trained in stage 1

[krisbian4]

Hii everyone
is there any update on this? I actually follow the instruction from here: https://www.youtube.com/watch?v=gMx2CApKhuo
In the video, he uses the Stage 1 (SFT) model. I tried the same, and it actually works.
But then if I follow the instruction from the repository

torchrun --standalone --nproc_per_node=4 train_reward_model.py
--pretrain "/path/to/LLaMa-7B/"
--model 'llama'
--strategy colossalai_zero2
--loss_fn 'log_exp'
--save_path 'rmstatic.pt' \

It is mentioned that i need to used the pretrain from base llama model, not the one I trained from stage 1 (Coati-7B).
Which one is actually the right implementation?