[Feature Request] Support Speaker Diarization

[uniqueness-ae]

Implement speaker diarization for the existing mlx whisper support to:

1. Enhance transcription accuracy in multi-speaker conversations
2. Distinguish between different speakers in the output
3. Improve overall usability of the transcription feature

This addition will provide more insightful and structured transcripts, making it easier to analyze and understand complex audio content. Thanks

[Hoohm]

Would love to see this as well.
I can help out in making the feature but I need some pointers as to how it would be possible.

[uniqueness-ae]

I tried pyannote.audio model using rented cloud GPUs and had some success. Perhaps if there is a way to run this mlx, it will probably run faster. Maybe even better if it’s coupled with whisper to simplify the process. There is a repo from m-bain called WhisperX that does this. Could help as a reference.

[krzysztoflupa]

@uniqueness-ae On how many speakers have you tried? WhisperX didnt work properly at all with more than 20 speakers still do not know why...