milestone: Ichigo v0.5 Multi-lingual

[hahuyhoang411]

Goal

Expand Ichigo model's capabilities to support multiple Asian languages (and more), with initial focus on Vietnamese and Singlish, achieving production-grade speech recognition and synthesis quality.

• planning: Train our own Quantizer for multilingual speech to work with Whisper Encoder #146

Description

Current State

• Ichigo model currently supports English language only
• Limited ability to handle Asian language phonemes and tonal variations

Proposed Solution

• Develop a multi-lingual speech foundation model based on WhisperSpeech approach
• Training Speech Encoder for Asian languages

Resources

Multi-lingual Base

• LibriSpeech (English baseline)
• Common Voice (Multi-language)
• GigaSpeech (Vie, Thai & Indo)
• FLEURS (1000+ hours, 102 languages)
• ... etc

Language-Specific

• Vietnamese

  • ViVoice (100+ hours)
  • BUD500 (500 hours)
  • ... etc

• Singlish

  • National Speech Corpus (NSC)

Reference:

• WhisperSpeech: https://github.com/collabora/WhisperSpeech
• PhoWhisper: https://github.com/VinAIResearch/PhoWhisper

Experiment Log

• Add Google Sheet here (public view-only)

Tasklist

• task: Pretraining dataset for Multilingual Quantizer training (Phase 1) #118
• planning: Train our own Quantizer for multilingual speech to work with Whisper Encoder #146
• task: Instruct Dataset Creation for Multilingual Speech (Phase 2) #121
• task: Ichigo LLM v0.5 Training #122
• task: Use VieTTS if output language is Vietnamese #123