Is it possible to use the Device also as a Bluetooth-Proxy?

[ChristophCaina]

Hi There,

The reason for me to buy some ESP32-S3-Box are pretty simple:
A higher "family acceptance factor" for a nice little device with a Display and so on, compared to an ESP32 Dev-Board.

So right now, the main purpose is to replace the existing Bluetooth-Proxies with ESP32-S3-Boxes.... but I would also like to use the Display (to show some information from HomeAssistant) - and - use the internal Voice-Recognition options together with HA's Assist.

For the Bluetooth-Proxy Stuff, I am using the Device within ESPHome right now... but I know, that many options might not be possible as of now with this method... so I am asking myself, if I still could use the Bluetooth-Proxy while running this project ... ?

Thanks and with best regards

[kristiankielhofner]

The "family acceptance factor" is one of the big reasons we started Willow!

Us makers and "geeks" have no problem with a solution that's rough around the edges, not aesthetically pleasing, and we'll even get excited about things that do cool stuff but actually don't really work that well (when compared to polished commercial products).

With Willow we're aiming to "have your cake and eat it too". Something that we can get excited about and do all kinds of interesting things with but the rest of the family doesn't know the difference between Willow and Alexa.

I had to read up on Bluetooth Proxy because I've never heard of it before.

We currently have Bluetooth disabled completely but that will change soon enough - things like pairing with external speakers for audio output, potential initial configuration, etc.

That said, I'm sure people can understand that we are already doing A LOT with a $50 microcontroller based device. Our first priority is pristine far-field audio with wake word, speech recognition, etc. We need to evaluate each new piece of functionality very carefully as we have to ration every single byte of memory and CPU cycles. Willow has grown quite a bit in it's first week of public life but we want to make sure we don't bloat it to the point it's unusable in six months, a year, etc. We plan for multi-year support of the devices people have purchased and thus need to consider each feature very carefully.

In short, I'm not opposed to this fundamentally but I'd want to hear more interest from the community to add it to the list of supported features.

[Ulrar]

As far as I'm concerned I'd use it, but only if it's "free". If it's too heavy and requires disabling some other useful component then I'd probably leave it disabled.

With the very low consumption of all these devices, I personally don't think this has to be supported by Willow, but it'd be nice. For example I leave bluetooth_proxy disabled on my NSPanels because they're already quite busy driving the screen, I wouldn't be shocked if it had to be the same for Willow which is already doing quite a bit !

Currently I'm using cheap esp32-doit-devkit boards with ESPHome and just the bluetooth_proxy enabled, in little 3d printed cases to protect them in case my toddler gets her hand on one. I just plug them in out of the way on top of cupboards or bookshelves and such, they're actually easy enough to hide away and have quite the range. That said since they're limited to 3 connections per device, having more of them is always good.

[sparkydave1981]

+1 for enabling the ESPhome style Bluetooth Proxy on these devices. I have a few ESP32's around the house with the intent of having whole home Bluetooth coverage back to my HA server. Being able to use these ESPbox's as part of the network would be excellent.

[ChristophCaina]

Some People are working on support for this box for ESPHome...
esphome/esphome#4793 (comment)

[sparkydave1981]

That would surely then remove the boxes ability to be a local voice activated assistant though since ESPhome doesn't (yet) have wake work ability. I would rather just run more ESP32's around the place than lose the Willow assistant feature.

[ChristophCaina]

Hi, sorry - I've posted the wrong link above...
esphome/feature-requests#2239

They are trying to enable wakeword detection, too... but I think the best option would be, if both projects could somehow work "together" and combine their powers to get the maximum out of this little device ;)

[kristiankielhofner]

This issue is we have fundamentally different goals and fundamentally different approaches. Even when esphome undertakes the significant amount of effort to support the hardware they still need to work through the myriad of additional frameworks and broader platform support (to support things as fundamental as wake word detection, voice activity detection, etc). From that thread I'm happy to see them take hints from Willow on their early work to support the ESP BOX hardware but they have a very long road ahead of them for feature parity. Assuming they get there (it's a very capable team and community, I'm sure they will) there are additional issues.

Willow aims to be platform agnostic - it already works with HA, openHAB, and generic HTTP/REST endpoints with more to come.

esphome is completely linked to and dependent on Home Assistant. This is ideal for their goals and makes perfect sense for their typical use cases. Both @stintel and I enjoy the use of MANY esphome devices with our HA installs - we truly love and appreciate HA/esphome and have a tremendous amount of respect for the team and community.

However, what really enables most of the functionality that makes the Willow experience possible is the broader architecture, including components like the Willow Inference Server. HA has support for STT and TTS but it is also significantly behind Willow/WIS in terms of end-user experience. We set out from the very beginning to provide an equivalent or superior Alexa-like end user experience. Open source, self-hosted Alexa that also matches or beats Alexa on every possible metric - no compromises whatsoever. With this lofty goal in mind we have approached voice almost completely differently in terms of fundamental architecture and design.

I have addressed this elsewhere but long story short I have twenty years of experience working with audio, speech, ML, and real-world high performance design and architecture for all of these disparate technologies. Without being critical or "throwing shade", my concern is the Home Assistant team (IMO) has made several overall architecture and design decisions that when combined together make a Willow-like experience fundamentally impossible within the HA ecosystem. I feel they will either need to revisit some of these fundamental design decisions or continue to offer a sub-par experience. I know this position may be controversial but over the decades I have made every possible mistake personally or watched others in the space do so - and learned A LOT in the process. Voice and speech is a fundamentally different animal and they seem to be attempting to bend these fundamental realities of voice to the HA ecosystem as opposed to adapting HA, esphome, etc for voice (to put it succinctly).