Dataset type conversion utilities #8
Labels
🏋 DDPO
Related to DDPO
🏋 DPO
Related to DPO
🏋 GKD
Related to GKD
🏋 Iterative SFT
Related to Iterative SFT
🏋 KTO
Related to KTO
🎯 optimal import sentence
❓ question
Seeking clarification or more information
System Info
System Info
Some things that are not really correct in the dataset type conversions in https://huggingface.co/docs/trl/main/en/dataset_formats#utilities-for-converting-dataset-types:
when converting a preference dataset to an unpaired preference dataset with unpair_preference_dataset(), we are converting from a relative ranking to an absolute ranking. In a preference dataset, despite having a "chosen" and a "rejected" example, both can be good or both bad, just one slightly better/worse. See the example below.
So one should not convert a Preference dataset to an Unpaired Preference Dataset without keeping an eye on absolute ratings from e.g. a reward model.
Suggestion: At least add a warning to the documentation and conversion code or even remove it
when converting from Unpaired preference or Stepwise supervision to anything un-labeled like Language modeling or Prompt-completion, only the good (label=True) examples should be used. Like when converting from a Preference dataset it only uses the chosen completions.
Suggestion: Can easily fix that in the example conversion code
Information
Tasks
examples
folderReproduction
from datasets import Dataset
dataset_dict = {
"prompt": ["The sky is", "The sun is"]
"chosen": [" blue.", " in our solar system"],
"rejected": [" above.", " in the sky."]
}
dataset = Dataset.from_dict(dataset_dict)
dataset = unpair_preference_dataset(dataset)
dataset[1]
Expected behavior
{'prompt': 'The sky is', 'completion': ' blue.', 'label': True}
{'prompt': 'The sky is', 'completion': ' above.', 'label': True}
Checklist
The text was updated successfully, but these errors were encountered: