Add Llama 3.1 Instruct Chat template, Ensure Correctness of 3, Small Refactor of Chat Template registration for shareGPT

[alpayariyak]

Description

• Add Llama 3.1 Instruct Chat template, which mainly differs in these ways:

  • it always has system prompt regardless of whether the conversation does
  • In the system message there's always a knowledge cutoff and today's date, user system prompt content is added after it

  To use it, specify chat_template: llama31 in the config.

• Ensure correctness of Llama 3 Chat template by removing the default assignment You are a helpful assistant system message.

• Refactor some redundancy out

Motivation and Context

Important for those who want to fine-tune the instruct versions, as well as anyone who just wants to have the same prompt template as instruct versions, even if they are fine-tuning from base.

How has this been tested?

Preprocessing combinations of the following 2 variables in a multi-turn setting:

• using llama3 vs llama31
• dataset with system prompts vs without
  With the debug flag, making sure that the loss mask is correct and that the tokens match using tokenizer.apply_chat_template

Screenshots (if appropriate)

Example Llama 3.1 tokenized output:

• When there is a system message in the conversation
  
• When there isn't
  

[winglian]

@alpayariyak pushed up some fixes to this branch. should probably re-test the functionality

[alpayariyak]

Just re-tested, looks good!

[alpayariyak]

There's a weird bug - if assistant content is only 1 token, it's masked out