Report is accessible at : https://docs.google.com/document/d/1-_UrFEylkIIsJ3JS4Fiv9_nquD0LjlEzGezDqQX1Cjk/edit?usp=sharing
Fine-tuned model is available at: Falcon 7B Mental Health LORA
Results available at: WandB - Fine Tuner
The goal of this project is to develop a prototype for the fine-tuning of the Falcon 7 billion LLM (Large Language Model) using a dataset focused on mental health conversations. The project utilizes LORA ( Low Rank Adaptation ) for fine-tuning the LLM, employing the Hugging Face Supervised Fine Tuning approach. Techniques such as Parameter Efficient Fine Tuning (PEFT) are incorporated.
Note: Results in the notebook may vary from those in the report.
- Model: Falcon-7B-Instruct
- Dataset: Mental Health Counseling Conversations
The model was trained using the Hugging Face LORA & PEFT libraries.
| Configuration / Training | Value |
|---|---|
| r | 8 |
| lora_alpha | 32 |
| LORA Modules | ['query_key_value', 'dense', 'dense_h_to_4h', 'dense_4h_to_h'] |
| lora_dropout | 0.05 |
| bias | "none" |
| per_device_train_batch_size | 16 |
| gradient_accumulation_steps | 16 |
| save_steps | 100 |
| learning_rate | 0.001 |
| max_grad_norm | 0.3 (maximum gradient norm for gradient clipping) |
| max_steps | 600 |
| warmup_ratio | 0.03 (number of steps for linear warmup) |
| sequence_length | 1024 |
| maximum steps | 600 |
| Training Time | 6 hours |