Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[audio] Add pcm audio websocket with dialog support #4032

Open
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

GiviMAD
Copy link
Member

@GiviMAD GiviMAD commented Jan 10, 2024

These PR adds a WebSocket Adapter that allows transferring PCM audio to a sink and source whose existence will be tied to the WebSocket connection (register/unregister the components on connection/disconnection) also allows to spawn a dialog processor instance connected to those (will also be tear down on disconnection).

The PR is incomplete and untested, I'll try to add a client to the UI and develop both PR in parallel.

Also IDK if placing the code in the voice bundle is correct, I did it there because it already requires the audio bundle. For me it makes more sense to do it on the audio bundle but I'll have to require both the voice and websocket bundles there.

Best regards!

@GiviMAD GiviMAD requested a review from a team as a code owner January 10, 2024 20:45
@kaikreuzer
Copy link
Member

Also IDK if placing the code in the voice bundle is correct

As the websocket adapter is for audio streams in general, I would say that it should ideally be placed in the audio bundle.
The dependency on the voice manager is very small; maybe you could come up with some small hook, which could be used by the voice bundle to register the dialog processing logic into the websocket adapter?

@GiviMAD
Copy link
Member Author

GiviMAD commented Jan 13, 2024

Also IDK if placing the code in the voice bundle is correct

As the websocket adapter is for audio streams in general, I would say that it should ideally be placed in the audio bundle. The dependency on the voice manager is very small; maybe you could come up with some small hook, which could be used by the voice bundle to register the dialog processing logic into the websocket adapter?

Yes, having it on the voice bundle is quite unintuitive, I haven't though on that solution seems like the correct one.

Thank you for the feedback!

@GiviMAD GiviMAD changed the title [WIP] [audio/voice] Add pcm audio websocket with dialog support [WIP] [audio] Add pcm audio websocket with dialog support Jan 14, 2024
Signed-off-by: Miguel Álvarez <[email protected]>
Signed-off-by: Miguel Álvarez <[email protected]>
Signed-off-by: Miguel Álvarez <[email protected]>
Signed-off-by: Miguel Álvarez <[email protected]>
@GiviMAD
Copy link
Member Author

GiviMAD commented Jan 14, 2024

I have to refactor the code into the websocket.io bundle for it to build without modifying the BOM features, but I think it also makes sense to have it there with the other WebSocketAdapter implementations.

Signed-off-by: Miguel Álvarez <[email protected]>
Signed-off-by: Miguel Álvarez <[email protected]>
@wborn wborn added the enhancement An enhancement or new feature of the Core label Jan 16, 2024
Signed-off-by: Miguel Álvarez <[email protected]>
@GiviMAD
Copy link
Member Author

GiviMAD commented Jan 16, 2024

I think these is ready for the review.

I was able to write a POC in the UI and everything seems to work.

A quick video recorded with my mobile:

video_2024-01-16.mp4

Still need a lot of work on the UI side, I'm going to create a linked issue there asking for some guidance, I wanted to first have a POC to know there are no big issues with the integration.

Best regards!

@GiviMAD GiviMAD changed the title [WIP] [audio] Add pcm audio websocket with dialog support [audio] Add pcm audio websocket with dialog support Jan 16, 2024
Signed-off-by: Miguel Álvarez <[email protected]>
@florian-h05
Copy link
Contributor

@openhab/core-maintainers Would be great if this could be reviewed for openHAB 5, as it would allow us to have a voice assistant inside the UI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An enhancement or new feature of the Core
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants