v0.0.4

Turbo-Alignment v0.0.4 Release Notes 🚀

What's New 😎

• 🚀 Performance Optimizations

  • Streamlined the processing of textual data by introducing Liger Kernels for Gemma2, significantly improving both computation time and memory management.
  • Switched the RM Trainer to use one concatenated forward pass instead of two, offering a more efficient training cycle especially with FSDP or Deepspeed.

• 🔄 Training Strategy Enhancements

  • Add precomputed-margin to pair-preference dataset to facilitate the application of algorithms like SLiC-HF with added support for DPO with margin.
  • Included a new feature for RM-Sampling to utilize multiple GPUs, accelerating the inference process.

• ✌️ New Losses And Metrics For Preference Optimization

  • Added APO-Zero and APO-Down losses, enriching the toolbox for preference optimization.
  • Added ASFT loss, which is effective approach that better aligns LLMs by optimizing absolute likelihood for each response
  • Integrated compute_flips metrics into DPOTrainer, providing more nuanced insight into model performance.

• 🔠 More Flexible Settings For SpecialTokensSetter

  • Introduced SpecialTokensSetting to better control all new tokens added to tokenizer and embedding model layer.

• 📀 Enhanced Dataset Handling

  • Added the ability to use not just bots but also assistant replicas in datasets.
  • Implemented functionality to skip system prompts in chat datasets.

• 🧹 New Logging Features

  • Added the ability to use ClearML logging.

Documentation and Tutorials 📚

• 📘 Update README, Docs and tutorials
  • Updated the README, documentation, and tutorials to provide clearer guidance to users, including a newly added citation section for academic referencing.

Improvements and Fixes 🛠️

• ⚙️ Dependencies Updates

  • Updated the versions of transformers, accelerate, and vllm to support modern architectures like LLama3.1 and Gemma2.
  • Enhanced project management with an updated poetry version, simplifying dependency resolution and packaging.
  • Removed AllenAI dependencies for a more streamlined package with fewer third-party requirements.

• 🧠 Corrected ORPO Loss

  • Added the missed NLL loss part in ORPOLoss.

• 🐙 vLLM Inference With Adapters

  • Added ability to use PEFT models with vLLM.

• 🥉 Fix Deepspeed Stage3 Problems

  • Added ability to train AutoModelForSequenceClassification with Deepspeed Stage3.

• 🐞 Tokenization Bugs

  • Addressed an error that caused VLLM to incorrectly use two tokens instead of one.
  • Implemented a fix for the keep_end truncation strategy in the chat dataset, ensuring text samples are correctly truncated.

Full Changelog 📝

You can view the complete list of changes in this release by visiting the changelog on GitHub: Full Changelog.

New Contributors 🌟

• @Myashka made their first contribution in #24

---

We hope you enjoy these updates! As always, we welcome your feedback and contributions to make Turbo-Alignment even better.

Don't forget to star ⭐️ the repo if you find it useful, and watch it for future updates.

Thank you for supporting Turbo-Alignment! 🙌

---

Need help or have questions? Reach out to us on GitHub Issues, and we’ll be there to support you.

---

Installation

Upgrade to the latest Turbo-Alignment release with:

pip install turbo-alignment==0.0.4

— Turbo-Alignment Team 🤫

v0.0.2

Turbo-Alignment v0.0.2 Release Notes 🚀

What's New 😎

• 🤗 Add SimPO and ORPO Trainers

  • Introducing the ORPO and SimPO trainers. These new additions allow you to experiment with cutting-edge preference optimization methods, which are doesn't require reference model.

• 🔧 Fix SLiC-HF Trainer

  • We've resolved issues that prevented the use of the SLiC-HF trainer.

Documentation and Tutorials 📚

• 📘 Add Multimodal Tutorial and Docs
  • Enhance your skills with our new tutorials and documentation designed for multimodal training pipelines.

Improvements and Fixes 🛠️

• 🐛 Fix Embeddings Initialization Strategy for GPT-NeoX

  • Token embeddings initialization now supports both LLama and GPT-NeoX architectures.

• 📈 Fix Multiple Logits in Chat Generator

  • The chat generator has been updated to handle multiple logits.

• 🔍 Fix Type of ID for Answer Message in rewards.py

  • Answer message IDs are now correctly typed as strings to match pydantic model expectations.

• 🛑 Add stop_strings into Chat Generator and Fix Multiple EOS Problem

  • Now, the chat generator manages multiple End-Of-Sequence tokens and incorporates using strings as EOS tokens.

• 🔄 .to('cpu') Per Batch into Chat Generator

  • We've optimized memory usage for batched chat generation by moving batches individually to the CPU.

• ✅ Fix model.config.use_cache When Not Using Grad Checkpointing

  • The KV-Cache now behaves predictably, ensuring proper functionality even in absence of gradient checkpointing.

Full Changelog 📝

You can view the complete list of changes in this release by visiting the changelog on GitHub: Full Changelog.

New Contributors 🌟

• @lmeribal made their first contribution in #8

---

We hope you enjoy these updates! As always, we welcome your feedback and contributions to make Turbo-Alignment even better.

Don't forget to star ⭐️ the repo if you find it useful, and watch it for future updates.

Thank you for supporting Turbo-Alignment! 🙌

---

Need help or have questions? Reach out to us on GitHub Issues, and we’ll be there to support you.

---

Installation

Upgrade to the latest Turbo-Alignment release with:

pip install turbo-alignment==0.0.2

— Turbo-Alignment Team 🤫

v.0.0.1 Release

Merge pull request #2 from turbo-llm/alekseymalakhov11-fix-docs

📝 Update README, docs & tutorials