Problem Launching Training for LLaMA-3 with KTO Strategy

[sklyar61]

Hello everyone. I am experiencing issues when attempting to train LLaMA-3 with the KTO strategy. Here are the configs I'm using:

Configurations:

rl: kto
remove_unused_columns: False
Datasets:

yaml

• path: "/path/to/all_data"
  data_files: "/path/to/orpo.jsonl"
  type: llama3.agrilla_chat
  split: train
  I am trying to preprocess data using the chat_templates(llama3) template and I receive the following error:

scss
axolotl/src/axolotl/utils/data/sft.py", line 689, in get_dataset_wrapper
raise ValueError(
ValueError: llama3.agrilla_chat
I attempted to change the data structure to:

json
{"conversations": [{"from": "system", "value": "..."}, {"from": "user", "value": "..."}, {"from": "assistant", "value": "chosen_answer"}, {"from": "assistant", "value": "rejected_answer"}]}
and modify the configs to:

yaml
rl: kto
remove_unused_columns: False
datasets:

• path: "/path/to/all_data"
  data_files: "/path/to/orpo.jsonl"
  chat_template: llama3
  type: chat_template
  This allowed me to run data preprocessing, but when launching the trainer, I encounter the following error:

less
Generating train split: 1228 examples [00:00, 75609.66 examples/s]
Traceback (most recent call last):
File "/axolotl/ax/lib/python3.10/site-packages/datasets/builder.py", line 2011, in _prepare_split_single
writer.write_table(table)
File "/axolotl/ax/lib/python3.10/site-packages/datasets/arrow_writer.py", line 585, in write_table
pa_table = table_cast(pa_table, self._schema)
File "/axolotl/ax/lib/python3.10/site-packages/datasets/table.py", line 2295, in table_cast
return cast_table_to_schema(table, schema)
File "/axolotl/ax/lib/python3.10/site-packages/datasets/table.py", line 2249, in cast_table_to_schema
raise CastError(
datasets.table.CastError: Couldn't cast
system: string
chosen: list<item: struct<role: string, content: string>>
child 0, item: struct<role: string, content: string>
child 0, role: string
child 1, content: string
rejected: list<item: struct<role: string, content: string>>
child 0, item: struct<role: string, content: string>
child 0, role: string
child 1, content: string
to
{'conversations': [{'from': Value(dtype='string', id=None), 'value': Value(dtype='string', id=None)}]}
because column names don't match
And so forth, leading to:

datasets.exceptions.DatasetGenerationCastError: An error occurred while generating the dataset

All the data files must have the same columns, but at some point, there are 3 new columns (chosen, system, rejected) and 1 missing column (conversations).

This happened while the json dataset builder was generating data using

/Рабочий стол/Data/LLm/llama3/all_data/orpo.jsonl

What structure should I adapt the template to in order to successfully launch training for LLaMA-3 with the KTO strategy, and what configurations should be used?