Possible to use for real-time / streaming tasks?

[davidhariri]

Is it possible to use whisper for streaming tasks (with syntax)? For example, would it be possible for whisper to be bound to a websocket of streaming PCM data packets?

[jongwook]

It doesn't support real-time per se, but you could build something similar by e.g. incrementally transcribing the audio every second.

[codeyourwayup]

how is the performance?

[juanmc2005]

I recently wrote a tutorial for streaming speaker-colored transcriptions combining whisper and diart (based on pyannote.audio models and supporting websockets).

I'm sure many improvements can be made 😃 Here's a demo:

good1.1.mp4

[JennPlothow]

.js-local-storage-resumable
proton_recovery_phrase.txt

[FOREVEREALIZE]

@JennPlothow
Hey, uh, that contains you Proton Mail recovery i believe. You shouldn't post that.

[PabloLION]

Is the character @JennPlothow generated by someone with AI? Looks like someone if faking a Brazilian girl:

I saw on his/her linked in:
I will give you text content, you will rewrite it and output that in a re-worded version of my text. Reword the text to convey the same meaning using different words and sentence structures. Avoiding plagiarism, improving the flow and readability of the text, and ensuring that the re-written content is unique and original. Keep the tone the same.
Keep the meaning the same. Make sure the re-written content's number of characters is exactly the same as the original text's number of characters. Do not alter the original structure and formatting outlined in any way. Only give me the output and nothing else.
Now, using the concepts above, re-write the following text. Respond in Portuguese:

[dangolbeeker]

Hello everybody I came across this api with the exact same thing in mind. I created an organization for this. iInterpret the envision is an app that can translate verbal speech in real time for phone calls or communication in person maybe with a bluetooth piece? This would be a blessing for some of my business in China to go from Mandarin to english seamlessly. If this interests you or anyone else reading this please join the organization and feel free to reach out to me via [email protected]
https://github.com/Iinterpret

[Geczy]

@jojeyh can you share your repo

[osasisorae]

@jojeyh I would appreciate to see your repo.

[rahulbansal16]

If using React, I was able to accomplish this roughly using the voice activity detector npm module @ricky0123/vad-react. It breaks up speech segments based on VAD and then sends audio chunk to Whisper API. So when a speaker starts talking the npm module begins recording the audio, when speaker stops and there is silence it will call onSpeechEnd with audio chunk, even has utils function that encodes into wav before sending to Whisper API.

So how does the sentence between two silences affect accuracy? Does having shorter sentences decrease the accuracy?

@joorjeh

[foomprep]

@rahulbansal16 It was a while since I used this package but I believe the npm module in question has a minimum time chunk required for accepting as valid speech segment, otherwise it considers it a spurious voice recognition and ignores.

[foomprep]

@rahulbansal16 In general, shorter audio segments in ASR will necessarily be less accurate as they have less signal (read context) to determine the right text. You might want to check out resources at https://github.com/ufal/whisper_streaming. Usually for a streaming situation you use a rolling window and update the text as you move the window.

[bquast]

Thank you collegues. Working at the International Telecommunication Union (United Nations Specialized Agency for ICT) we would really like this for our meetings.

ps. the title of the org for me is quite hard to read, 'l then i'

[dangolbeeker]

Awesome look forward to working with you I took your suggestion and have renamed it to myLinguist.
The names not set in stone so open to other suggestions if you have a better name idea in mind.

[ggerganov]

Here is another attempt for real-time streaming:

rt_esl_csgo_1.mp4

This is using small.en on MacBook M1 Pro with 3 seconds audio step. Runs entirely on the CPU.
The repo is here: https://github.com/ggerganov/whisper.cpp

[chidiwilliams]

Such a cool demo. I made a similar real-time app (+ a GUI) by splitting the audio into chunks.

Demo: https://www.loom.com/share/564b753eb4d44b55b985b8abd26b55f7?t=34

Repo: https://github.com/chidiwilliams/buzz

[ggerganov]

And here is another example with even more "real-time" transcription using base.en:

rt_esl_csgo_2.mp4

[abdulhadyabas2]

link code please.

[odinho]

link code please.

It was in the main post.
Here you go: https://github.com/ggerganov/whisper.cpp/tree/master/examples/stream

[saharmor]

I've built Whisper Playground for developers to easily build real-time speech2text web apps
https://github.com/saharmor/whisper-playground

Whisper.Playground.mp4

[ibnbayo]

"Building wheel for pyaudio (pyproject.toml) ... error"
This is the error I keep on getting. I've tried installing it manually using brew, but it didn't work still.

[deadlinecode]

@ibnbayo
You need to install the dev version of pyaudio/portaudio

[shekharlilly]

no module named whisper. i'm using python 3.10.7

[makaveli10]

@shekharlilly try this, it works in chrome extension, Mozilla and also from microphone
https://github.com/collabora/whisper-live

[chaima-bd]

I did the installation for whisper by this 'pip install git+https://github.com/openai/whisper.git ' .
But when i work with pycharm or vscode I got this problem "ModuleNotFoundError: No module named 'whisper'"
any idea for solve this prob plz

[deadlinecode]

Mby check your python version
3.10 worked for me

[appvoid]

You can check my project: https://github.com/appvoid/vosper
It uses vosk for VAD and user feedback while uses Whisper in the back for the actual transcription.

[nyadla-sys]

Please see my project below, which uses the Whisper Tiny Tflite Model to implement audio streaming..
Using the TFLite model to Stream

[zanjabil2502]

If you want reduce processing time of transcribe when you use whisper for streaming, you can use whisper decoder for get only tokens of transcribe and decode it using tokenizer.

Because the buffer of audio from the streaming chunk dont have length until 30 second, and in the transcribe of whisper there temperature and logprob, and the other prob for get the best result of transcribe, it process will need more iteration, it means you will need time more longer

[zanjabil2502]

You cant use whiper c-translate, for 1 worker, this model have RTF 0.05-0.06 for large model, this RTF will incrase if you use multi concurrent. For large model, 5 minutes audio, i get RTF 0.6 for 15 thread concurrent. GPU memory used only 3.5 gb when you set 1 thread in the parameter and CPU core can reach 15 until 50 cores. But you can limit your core until minimum 2 cores, because if you set only 1 core, the process will longer 30 seconds.

[zanjabil2502]

This repo for whisper c-translate
Whisper C-Translate

[chobe]

This is very helpful. Thanks

[Mijawel]

When you use CTranslate2 and Silero, are you simply waiting for an audio gap then sending it to faster_whipser for transcription and using the model.transcribe function?

Or are you using a different (maybe lower level function like generate) in CTranslate2 faster_whisper>?

[zanjabil2502]

@Mijawel I waiting audio segment from VAD Silero. So VAD Silero will create audio segment from audio buffer, and it will transcribe using ctranslate2 model.transcribe.

For the VAD, i use timestamp method from VAD Silero. VAD will create list of timestamp so incoming audio chunk will be buffered and entered into VAD, when the length of list change, you can take the first array of list timestamp for entered into transcribe model.

[chengsokdara]

This is my take for React.js

useWhisper React hook can now do real-time transcription.

Repo:
https://github.com/chengsokdara/use-whisper

Demo: (Whisper seems to can not understand my accent 😅)

use-whisper-real-time-transcription.mp4

[FerLuisxd]

This is using the openai whisper api, right?

[mustafatofur]

@FerLuisxd you might want to consider using the Speech Recognition API for this. However, please note that not all browsers support it. You can find more information here: https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecognition

[diyism]

In essence, what we ultimately need is a Real-time Syllable Recognition engine with a mechanical keyboard precision,
because we can send the syllable sequence to LLM for example ChatGPT-4,
LLM will give us the best understanding and translation of the Syllable Sequence in the world,

for example for mandarin, I get the syllables of "ni3 hao3 ren2 men2 zong1 guo2 han4 zi4" and send it to ChatGPT-4,
the ChatGPT-4 can response the right characters of the best semantic analysis.

if we have a concise Real-time Syllable Recognition engine, LLM will replace the entire speech recognition industry.

ref: "Transcribe to IPA" is very important for realtime interaction application #318 (comment)
ref: [Feature Request] a concise model only output IPA/Pinyin syllables k2-fsa/sherpa-ncnn#177

[diyism]

Post App World requires a Syllable based Recognition Engine with a mechanical keyboard precision and with the speed which is faster than the mankind pronunciation.

Daniel Povey, the developer of Kaldi 2, believes that "Proactive Mandarin Syllable Recognition Engine" can't be achieved,but I still feel it can.
(ref: k2-fsa/sherpa-ncnn#177 (comment))

the ASR engines should focus more on the recognition of syllables.
while the task of analyzing vocabulary and sentences should be given to large language models.

[Gldkslfmsd]

Hi guys, I implemented realtime Whisper streaming for long audios in Python. Going to share it soon.

[Gldkslfmsd]

https://github.com/ufal/whisper_streaming

[diyism]

I've test your project with "python3 whisper_online.py nihao.wav --language zh --model small --min-chunk-size 0.5":
last processed 0.50 s, now is 2.82, the latency is 2.32
last processed 2.82 s, now is 5.20, the latency is 2.38
last processed 5.20 s, now is 8.05, the latency is 2.85
it seems the latency is too high.

Typically every syllable cost the same fixed 0.3 second in mandarin,
and a syllable only consist of "a consonant + a vowel" or "a vowel" without any tail consonants,
There are only 1300 mandarin syllables with tone in total.

[lrq3000]

This is a truly streaming model, not a hack in the wrapper, here is the paper the authors wrote along the code release: https://www.researchgate.net/publication/372684083_Turning_Whisper_into_Real-Time_Transcription_System

[ZQ-Dev8]

Has anyone seen or implemented a solution that can transcribe and translate from english into another language in real-time or with slight delay? Many of the projects here are great but I'm not seeing the English -> Other Language functionality anywhere.

[zanjabil2502]

Using whisper faster from c-translate and vad from silero.

[jnnnnn]

Whisper was trained to output either the spoken language or an english translation. It was not trained to translate to any language.

https://openai.com/research/whisper

In order to translate to languages other than English, consider using a separate model for the translation, such as Firefox's translate (Project Bergamot):

https://github.com/jelmervdl/translatelocally-web-ext

Otherwise you will have to train your own whisper model (which others are doing, see https://huggingface.co/models?pipeline_tag=automatic-speech-recognition&sort=trending&search=whisper ).

[gkorepanov]

Hi, I have made a small wrapper around OpenAI whisper API which adds kind of "streaming" capability to the API
https://github.com/gkorepanov/whisper-stream

It can be useful if you want to use existing API instead of running your own Whisper instance.

It splits the input audio into chunks of 30s each and sends them one-by-one to the API, which leads to much faster initial response and streaming experience for use cases where speed is important. It can be pretty easily extended for audio streaming applications as well, though it will not be real-time (expect around 40s latency when using such approach, or may be less if you reduce the chunk size).

[nalbion]

I've created a streaming whisper_server which sends audio from your mic through Whisper and streams as Server Sent Events or gRPC
https://github.com/nalbion/whisper-server

[makaveli10]

A nearly-live implementation to transcribe anything from audio in your browser to audio from your microphone.
https://github.com/collabora/whisper-live

[saharmor]

@makaveli10 that's cool! Would love to team up and add VAD + real-time transcription using websockets to Whisper Playground

[phineas-pta]

have anyone compare accuracy of whisper vs wav2vec2 for live transcription ? from my understanding whisper needs to pad audio to 30s so 1-2s chunks may not suitable, maybe wav2vec2 offer better accuracy for short chunks

[glangford]

Haven't tried this myself but here is an excerpt from one comparison regarding short transcripts (not necessarily live), fyi

"Like wav2vec, Whisper also exhibits a substantial degradation in mean WER per file on Conversational AI, Phone call, and Meeting data indicating pathological behavior on a subset of small files. As far as the normalization scheme, we find that Whisper normalization produces far lower WERs on almost all domains and metrics. This result is qualitatively similar to the results of the original Whisper paper."

Benchmarking Top Open Source Speech Recognition Models: Whisper, Facebook wav2vec2, and Kaldi
https://deepgram.com/learn/benchmarking-top-open-source-speech-models

[Alireza29675]

If you need real-time Whisper transcription in the browser, check out my TypeScript package whisper-live. It's framework-agnostic, uses the OpenAI Whisper model for live transcription and is easy to integrate.

📦 Install with:

npm install whisper-live

More details here: https://github.com/Alireza29675/whisper-live

Happy to help if you have any questions!

[egorsmkv]

I have found https://arxiv.org/abs/2406.10052 a nice solution to solve streaming whisper.

[Kishlay-notabot]

Hi! Is there any repo which has real-time transcribing using any model [english or non english] which uses VAD to split chunks? I can't seem to find one with python as the core lang.

[Gldkslfmsd]

yes - https://github.com/ufal/whisper_streaming

[Kishlay-notabot]

this requires a input wav file. I want to do it using my laptop's microphone in real time.

[Gldkslfmsd]

It doesn't. Use server, read the README. Kishlay Kisu ***@***.***> schrieb am Do. 15. 8. 2024 um 18:28:

…

this requires a input wav file. I want to do it using my laptop's microphone in real time. — Reply to this email directly, view it on GitHub <#2 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABIRQXKSZYEAQIVOEQBDH5DZRTJKDAVCNFSM6AAAAAAQSII5EWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMZVGAYDANQ> . You are receiving this because you commented.Message ID: ***@***.***>

[Kishlay-notabot]

I'll check that out, thanks

[Kishlay-notabot]

@Gldkslfmsd I checked out whisperlive, vosper, whisper-server, whisper-streaming and whisper-playground.
Whisper-server uses OpenAI's API, but i am trying to do it with local inference. And I think whisperLive is what i need but I have to try it out. I can't comprehend the association of whisper with frontend in the whisper-playground project. I'll try whisper streaming and whisperlive out now.

[paschaldev]

You can try this with transformer.js. Works with browser that has support for WebGPU (Chrome browser)

https://github.com/xenova/transformers.js/tree/v3/examples/webgpu-whisper

[LucasAssis00]

Hello, guys. Does anyone use whisper in a project that transcribes small chunks of audio per turn? I was using speech_recognition library to do something like this but I need a whisper trained model cause it involves portuguese medical jargons, so the default whisper does not work so well even with the large model.
To do it asynchronously I am using pyaudio to record the wav files, but I cannot think a way of merging it to do synchronously.

[PabloLION]

i have a repo that split audios tho the performance is not guaranteed. You can see if the code helps.

[Slepetys]

Oi Lucas,
Check this:https://github.com/Dadangdut33/Speech-Translate, I am having good results with it.
It automatically split the audio input, that you can select either from the microphone or from the speaker and transcribes it using fast-whisper or whisper. It is not difficult to change the source-code to join the microphone and speaker audio into a single input, if needed.

What it does not implements is diarization, which can be done using the same building blocks as whisper-diarization

All the best