Release 2.4.0

Release 2.4.0

This release incorporates a third evaluation task (Fineweb) into the current competition starting on block 4,250,808. At this time the weighting of each dataset will be 90% MMLU, 5% Word Sorting, and 5% Fineweb.

Subnet

• Added new Fineweb evaluation task.
  • This evaluation scores models based on the computed average cross entropy loss on samples from Fineweb.
  • It is the same evaluation from subnet 9. Including it helps ensure the finetuned models do not lose too much of their original context.
  • Includes a check to make sure models are generating reasonable output. Meaning they are not too repetitive within or across responses.
• Improved definition of the competition schedule to include eval tasks.
  • This makes it easier to add new evaluations to competitions at specific weights and makes it easier to view them as a miner.
  • See COMPETITION_SCHEDULE_BY_BLOCK in constants/__init.py__ to view for yourself.

Validators

• Improved the logic around strategy selection for sharing files across subprocess boundaries. This will help avoid overflowing /dev/shm.

Miners

• The new dataset loader for the fineweb task can be found at https://github.com/macrocosm-os/finetuning/blob/main/finetune/datasets/hugging_face/hugging_face_loader.py.

  • As mentioned this will be incorporated into the existing competition starting in block 4,250,808 so please take this into consideration for your training.
  • Note that this supports general hugging face datasets. Currently constants are included for Falcon and Fineweb. The current competition is only using Fineweb data.

  Validators should update as soon as they can. Note that due to requirement version updates you will need to rerun
  python -m pip install -e .