A question about rlootrainer

[macheng6]

Method description

rlootrainer does not seem to use the self.policy model in train() function. I don't know the meaning of self.policy in the init function.

Open source status

• The method implementation is available
• The model weights are available
• The training datasets are available

Provide useful links for the implementation

No response

[asparius]

It utilizes self.model, which is defined in [this line](

trl/trl/trainer/rloo_trainer.py

Line 162 in 6d4ed07

self.model = policy

). This approach is also adopted in PPOTrainer. I believe this is a deliberate nomenclature choice, designed to remain consistent across various preference learning frameworks without introducing the complexity of aligning with the diverse terminologies used in academic papers.