You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Model-based evaluation is at the heart of successful model development -- as a reward model for training, and as a replacement for human evaluation. To train such evaluators, the standard approach is to collect a large amount of human preference judgments over model responses, which is costly and the data becomes stale as models improve.
Describe the solution you'd like
integrate meta's self taught evaluator as a packaged step
Is your feature request related to a problem? Please describe.
Model-based evaluation is at the heart of successful model development -- as a reward model for training, and as a replacement for human evaluation. To train such evaluators, the standard approach is to collect a large amount of human preference judgments over model responses, which is costly and the data becomes stale as models improve.
Describe the solution you'd like
integrate meta's self taught evaluator as a packaged step
Additional context
repo : https://github.com/facebookresearch/RAM/tree/main/projects/self_taught_evaluator
paper : https://arxiv.org/abs/2408.02666
The text was updated successfully, but these errors were encountered: