[Thanks]

[cocoon]

As there is Diskussions are not enabled, I thought I "abuse" the Issues to say: Thank you!

I just play around at home currently and wanted to see how far I get using rust to get:

• Speech to Text
• Text to intent
• automate stuff

As I already have the last thing (automate) running since longer time but using Alexa, custom skill and AWS, I thought, let's see what is out there to have all running local and best all in rust. (I am not really good with rust, but I still like it).

I first found the more or less abandoned sonos snips-nlu project, but I really got everything compiled and running in the end.
Then I found vosk and also got it compiled and running.

The last missing thing was I wanted to have a websocket server that receives audio and translates it into text that can be used to get intents out of it.

This is where I found your posts and this repository.

Today I successfully integrated (or injected) Vosk into your already existing audio receiving here:

streamer-template/src/axum_server.rs

Line 163 in ec2586c

Message::Binary(audio) => {

collecting the incoming data and use the WavWriter to recreate the audio file in memory.

reuse your audio decoding sample recreation code and could get Vosk to extract words out of the audio data (using https://ttsmp3.com/ to generate spoken text and using audacity to export the mp3 as Wav). 🚀

For developing I have all compiled under windows and plan to compile it later for armv7-unknown-linux-musleabihf.

I had some crashes / disconnects with the client I think here:

streamer-template/src/bin/client.rs

Line 151 in ec2586c

while let Some(res) = ws_rx.next().await {

and modified it to this and had no crashes anymore, but don't know if this is any good:

use futures::{SinkExt, StreamExt, FutureExt};
...

  while let Some(res_option) = ws_rx.next().now_or_never() {
        match res_option {
            Some(res) => {
                let res_m = res?;
                if let Message::Text(res_m) = res_m {
                    if first_instant.is_none() {
                        first_instant = Some(Instant::now().duration_since(start_instant));
                    }
                    info!("Got message: {}", res_m);
                }
            }
            
            None => info!("Connection closed?"),
        }
    }

[xd009642]

Thanks for the kind words, glad to see it's proving a useful resource 🎉

That change looks strange, because it seems the moment you don't have a message immediately available it will exit the receiving loop and then it will go to the cleanup logic and should disconnect after that. I would expect that code to disconnect more and cause more issues 🤔

If you can share more details of the crashes I can look into it

[cocoon]

If you can share more details of the crashes I can look into it

Great thanks, will try to collect some details and compile a fresh clone the next days.
It was some OS error ... maybe it only happens on windows.

[cocoon]

OK, got it reproducible.

I am running it on Windows 11.

I did a fresh clone today, followings simply the example commands on the main page here.

Opened 2 Terminal Windows running these commands:
cargo run --release
cls && cargo run --release --bin client -- -i x:\xyz\xyz.wav --addr ws://localhost:8080/api/v1/simple

resulting in client error:

Error: IO error: Eine bestehende Verbindung wurde softwaregesteuert
durch den Hostcomputer abgebrochen. (os error 10053)

Caused by:
    Eine bestehende Verbindung wurde softwaregesteuert
    durch den Hostcomputer abgebrochen. (os error 10053)
error: process didn't exit successfully: `target\x86_64-pc-windows-msvc\release\client.exe -i x:\xyz\xyz.wav --addr ws://localhost:8080/api/v1/simple` (exit code: 1)

And if setting the log level to Debug or Trace, the client process never finishes.
set RUST_LOG=Trace

[xd009642]

So if the server hasn't bound to the port by the time the client starts you'll get an error like that. Can you wait until /api/v1/health returns healthy from the server then run the client process?

[cocoon]

the server is running fine and the client is sending, it seems to happen near the end of the sending from the client.
Can't you reproduce it?

[xd009642]

I can't reproduce it on Linux and don't have a windows machine unfortunately

Hmmm the stop message in the client is set to disconnect: true. This is because after the audio file is sent we don't want to send anything else and we don't want to faff around trying to do timeouts or something to guesstimate when the server has to do no processing and get the client to trigger the disconnect.

In linux we get a close frame etc and everything is fine, but maybe that's not the case in windows or the windows impl handles it a bit worse 🤔

One way you can solve this is - as you're using the simple endpoint you can keep a track of the end time and when it's within so much of the file duration (to account for potential rounding errors), just ignore any errors from the Rust. If you were using the segmented one you could use the is_final: true and if you've seen one of them or gotten no messages when the error happens just ignore it.

I'll make a note to double check some of the websocket stuff, if I go through and update to latest axum and tungstenite I guess there's a chance it might go away there's been some changes on both 👀

[cocoon]

maybe that's not the case in windows or the windows impl handles it a bit worse

Yes, that is what I thought.

as you're using the simple endpoint
I have not decided (or really thought about it) what I want or need to use in the end, it was just the first one I used to try to get vosk working at all. But for intents I think I just need a full sentence in the end.

I of course already tried to search the internet for the error and I found other reports of it, but the only "solution" was the one I showed at the beginning. But it is not so easy.

It just doesn't feel good to not know what the real problem is there and what the proper handling would be.
I will try to experiment, maybe I find something.

looks interesting:
microsoft/Windows-Containers#114 (comment)
microsoft/Windows-Containers#114 (comment)

[cocoon]

As I found out that tungstenite doesn't support proxies, I added this now:
https://chuxi.github.io/posts/websocket/

and let it run through Burp proxy ... and ... no crash 🤣
UPDATE: But only once ... with additional runs I got the crash again.

Looks like a timing problem, if I set the client and server to
RUST_LOG=Info
I get the crash.

If I set it to:
RUST_LOG=None

No crash.

If you want to add proxy support, especially for debugging, it works.

async fn run_client(args: Cli) -> anyhow::Result<()> {
        info!("Connecting to: {}", args.addr);

        //let proxy_url = "socks5://127.0.0.1:1080";
        let proxy_url = "http://127.0.0.1:8088";
      
        let proxy = streamer_template::proxy::InnerProxy::from_proxy_str(proxy_url)
            .expect("failed to parse inner proxy");

        let mut tcp_stream = proxy.connect_async(&args.addr).await
            .unwrap_or_else(|e| panic!("failed to create proxy stream: {}", e));


        let (ws, _) = match client_async_tls(
            &args.addr, tcp_stream).await
        {
            Ok(ws) => ws,
            Err(_e) => {
                error!("Timed out trying to connect to socket");
                anyhow::bail!("Timed out trying to connect");
            }
        };

    info!("Connected to server, sending packets");

    let (mut ws_tx, mut ws_rx) = ws.split();

[cocoon]

Commenting out the trace here also prevents it to crash:

streamer-template/src/bin/client.rs

Line 49 in 2287c6c

let _guard = setup_logging().expect("Failed to setup logging");

//let _guard = setup_logging().expect("Failed to setup logging");