README.md

Finetuning Llama 3

It's easy to finetue Llama 3 on the question & answer dataset we just created.

./scripts/train.sh

The python is as follows

lamini-examples/06_ift/train.py

Lines 1 to 48 in 03f35d3

	from lamini import Lamini
	import jsonlines
	def main():
	llm = Lamini(model_name="meta-llama/Meta-Llama-3-8B-Instruct")
	dataset = list(load_training_data()) * 10
	llm.train(
	data_or_dataset_id=dataset,
	finetune_args={
	"max_steps": 300,
	"early_stopping": False,
	"load_best_model_at_end": False,
	},
	)
	def load_training_data():
	path = "/app/lamini-earnings-sdk/data/results/generated_q_a.jsonl"
	limit = 10
	with jsonlines.open(path) as reader:
	for index, obj in enumerate(reader):
	if index >= limit:
	break
	header = "<|begin_of_text|><|start_header_id|>user<|end_header_id|>"
	header_end = "<|eot_id|><|start_header_id|>assistant<|end_header_id|>"
	yield {
	"input": header + make_question(obj) + header_end,
	"output": obj["answer"] + "<|eot_id|>",
	}
	def make_question(obj):
	question = (
	f"Consider the following company: {obj['ticker']} and quarter: {obj['q']}. "
	)
	question += obj["question"]
	return question
	main()

Monitoring your job

After you submit a training job, it is scheduled on the cluster. You can monitor the progress of the job by visiting the link provided in the output of the training script.

python3 /app/lamini-earnings-sdk/06_fine_tuning/scripts/../train.py

Uploading data....
Upload to blob completed for data.
Data pairs uploaded to blob.

Your dataset id is: 0713f8cfa5746a0897079e7f249a048deb653cf7e849d6fc26f3d2dacc5722d0 . Consider using this in the future to train using the same data.
Eg: llm.train(dataset_id='0713f8cfa5746a0897079e7f249a048deb653cf7e849d6fc26f3d2dacc5722d0')
Training job submitted! Check status of job 6367 here: https://app.lamini.ai/train/6367

The page lets you monitor all of your jobs, view eval results, view loss curves, and logs. Jobs will automatically use all of the GPUs in your Lamini cluster.