Multi languages. #2391
-
|
Hello everyone, I'm testing Whisper using an audio file with multiple languages and would like to know if it's possible to have it return the detected language for each segment. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 5 replies
-
|
whisper doesn't support audio with multiple lang |
Beta Was this translation helpful? Give feedback.
-
|
the first lines of the readme talks about multilingual. |
Beta Was this translation helpful? Give feedback.
-
|
multilingual as monolingual audio but from different languages, not as audio with multiple lang |
Beta Was this translation helpful? Give feedback.
-
|
You're wrong. I tested audio in multiple languages several times, and the transcription was accurate, which is why I mentioned it in my first message. I recommend you test it. |
Beta Was this translation helpful? Give feedback.
-
|
@guilhermeasena32 While it is true that Whisper can sometimes transcribe multiple languages in the same audio, it won't do this reliably. Whisper was trained on monolingual audio files for a range of separate languages, but probably some examples of multilingual audio files were included but incorrectly labelled as single language audio files. As a result, Whisper can sometimes output a transcript for a multilingual audio with full multilingual transcripts, but incorrectly labelling it as a single language. Even if you are OK with the incorrect labelling, the problem is that Whisper's training data just didn't include enough examples of multiple languages in the same audio file, which is why it sometimes works and sometimes fails. At the same time, Whisper was also trained on the translation task, and so what will happen sometimes is that when Whisper is expecting the audio file to contain a single language, and yet it contains multiple languages, Whisper may end up translating the other language instead of code switching and transcribing the other language. (Although it would be interesting to test whether this is less of an issue with the turbo model which wasn't trained on translation...) |
Beta Was this translation helpful? Give feedback.
-
|
@ryanheise Thank you for the explanation. |
Beta Was this translation helpful? Give feedback.
@guilhermeasena32 While it is true that Whisper can sometimes transcribe multiple languages in the same audio, it won't do this reliably. Whisper was trained on monolingual audio files for a range of separate languages, but probably some examples of multilingual audio files were included but incorrectly labelled as single language audio files. As a result, Whisper can sometimes output a transcript for a multilingual audio with full multilingual transcripts, but incorrectly labelling it as a single language.
Even if you are OK with the incorrect labelling, the problem is that Whisper's training data just didn't include enough examples of multiple languages in the same audio file, which is …