Skip to content

Commit

Permalink
Fix ruff formatting
Browse files Browse the repository at this point in the history
Signed-off-by: Mustafa Eyceoz <[email protected]>
  • Loading branch information
Maxusmusti committed Oct 17, 2024
1 parent 4d095c3 commit a45e82b
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/instructlab/training/data_process.py
Original file line number Diff line number Diff line change
Expand Up @@ -356,7 +356,7 @@ def main(args: DataProcessArgs):
data_with_labels = data_with_labels.select_columns(["labels", "input_ids", "len"])
max_id = len(tokenizer) - 3
final_valid_data = data_with_labels.filter(
lambda x: all(tk < max_id for tk in x['labels']), num_proc=NUM_PROC
lambda x: all(tk < max_id for tk in x["labels"]), num_proc=NUM_PROC
)
if len(final_valid_data) < len(data_with_labels):
dropped_samples = len(data_with_labels) - len(final_valid_data)
Expand Down

0 comments on commit a45e82b

Please sign in to comment.