Switch to anyio (at least internally)

[frederikaalund]

Back in October 2020, I discovered anyio. It's basically a wrapper around several event loops: asyncio, trio, curio. This way, our library could cover more ground and benefit from structured concurrency.

Fill disclosure: I'm a proponent of structured concurrency. This is something that trio implements really well. In an ideal world, we would have trio in the Python standard library instead of asyncio. Unfortunately, this is not how history wanted it (trio came too late to the party). Today, asyncio is the de facto event loop implementation in Python.

Around mid-2020, I tried to take some of the ideas from structured concurrency and implement them on top of asyncio. I wanted the best of both worlds: Structured concurrency on top of the de facto event loop implementation.
Turns out I wasn't the only one with this idea: anyio beat me too it. anyio takes the ideas of structured concurrency and makes it available on top of asyncio (and other event loops).

I was really excited to find anyio and I quickly tried to port asyncio-mqtt to anyio. Unfortunately, I rant into some show stoppers in the anyio API at the time.

Recently (like 3 days ago) anyio fixed those show stoppers. The fix is still only available on master but it should land in a release soonish.

Sooo, what do you (the maintainers/community) think? Is structured concurrency a fad or the next big thing? Do we want to add anyio as a dependency (along with its dependencies)? Should we instead keep asyncio-mqtt lean and stick to the single-dependency (paho-mqtt)?

Let me know. :)

[flyte]

Typical! I just feel like I'm getting comfortable with asyncio and another paradigm shows up 😁

Without delving deeper into the details, it's hard for me to have a valid opinion, but from a surface level it sounds like a good idea.

On a personal level, I'm interested in learning how to make async easier and less error-prone, as well as catching up with the state of the art. I'm wary of using asyncio-mqtt as a whipping-boy to learn it with, but at the same time, there's no better way to learn something than by actually using it.

[flyte]

Also, what would this mean for the public API of the client? Would it require users to change how they use it, or would the change be invisible, and only internal as you allude to in the discussion topic?

[frederikaalund]

I can definitely recommend to read up a bit on structured concurrency. I think we'll see that term used a lot more in the future. :))

Would it require users to change how they use it, or would the change be invisible, and only internal as you allude to in the discussion topic?

Good question. 👍 I think that we can pull this of without breaking the API. I have a work-in-progress anyio-based port lying around somewhere. I basically took asyncio-mqtt and changed the internal event loop logic. I never finished it though, due to the show stoppers mentioned in the original post.

In any case, it is a major internal rework and there will be issues due to this. If we go in this direction, we'll just have to fix those as we go along.

I'm mostly concerned about bringing anyio in as a dependency (along with its set of dependencies). What will people think of that? Is asyncio-mqtt no longer "lean" now that it has multiple dependencies?

Also, suddenly the name "asyncio-mqtt" doesn't really apply anymore. The code would work on any event loop. That's also why I'd like to rename the whole thing (see #50).

Alternatively, I can fork it as "anyio-mqtt". It's just that I really don't want to fragment this "async MQTT library" space any further. :))

[flyte]

Well, I read the articles you linked to and I'm sold on "structured concurrency". Not least because I've spent two days building and debugging a really small but disproportionately frustrating asyncio-based game server process manager with some MQTT cherries on top.

I really like the look of trio, not least because the documentation is so much better than asyncio's. The trio implementation (trio-paho-mqtt) of what we're doing here is embarrassingly simple (albeit perhaps missing some of the features) and I "want me some of that". From what I've seen, AnyIO provides a similar API, so I'm enthusiastic about that too.

Since trio-paho-mqtt is so simple, I'd just start from scratch with anyio(-paho)-mqtt, which I suspect is what you did anyway, and then deprecate/replace this client once it reaches parity with it. That way you're only temporarily further fragmenting the MQTT client space.

[frederikaalund]

Glad to hear that. :) Let me see if I can find that anyio-based port I made a couple of months back.

[frederikaalund]

I made a work-in-progress anyio port in a new branch: feat-anyio. This was the easiest for me; we can move to a new repo later.

See the anyio_example.py file for details.

[0x0a11c0de]

My opinion is this library should only require asyncio. If it supports anyio, cool, but it should not require it.

[frederikaalund]

Hereby noted. 👍 Thanks for the feedback. I'll strive to make the anyio part optional so that you can use asyncio-mqtt as if it was just plain old asyncio. :)

[resingm]

If I am allowed to contribute to the discussion, I am opposed to the change. asyncio-mqtt makes it easy to write async MQTT applications with very few dependencies. In fact, it is just paho. I'd rather keep it that way. All other MQTT libs I found so far, are either not as mature and reliable or way larger than this lib.

[frederikaalund]

Of course, this is a call to everybody for input. The more feedback we get, the better. 👍

Indeed, anyio would be another dependency. That results in an overall "heavier" package. I share your concern here. Right now, asyncio-mqtt feels rather "light", which is a good thing. I'd hate to change that.

The maintainability of an anyio-based code base is just so much easier than a pure asyncio-based code base. I can't ignore that fact. Maybe we should aim for an anyio-based fork instead. My concern is that this effectively deprecates asyncio-mqtt, as we focus the development effort on the fork.

In any case, thanks for the feedback. 👍

[flyte]

I made my own rough take on the AnyIO Paho wrapper at the weekend. Just wanted to use it as a learning experience and I needed something for one of my projects.

Now, I've registered anyio-mqtt on PyPI just so we have the name, but I'm absolutely not planning on camping on it. My interest purely lies in ending up with a decent AnyIO MQTT client, so whatever ends up being the best one can live there. If anybody starts seriously using my version from PyPI even though it has no readme/description then that's sort of on them 😅

[flyte]

During creation of this, it occurred to me that it's quite possibly impossible to wrap Paho in a way that doesn't use threads unless Paho can allow us to handle socket creation ourselves. I also encountered quite a few issues with deadlocks thanks to all the mutexes used by Paho. Of course I fantasised about writing my own client (possibly using the Paho C library to handle the protocol, or just implementing the MQTT v5 myself), but it's not really a project I can dedicate enough time to.

[frederikaalund]

Great to hear that. 👍 I can barely find the time to keep asyncio-mqtt up to date, so feel free to take ownership of anyio-mqtt. I can help co-maintain if you want. :)

it occurred to me that it's quite possibly impossible to wrap Paho in a way that doesn't use threads unless Paho can allow us to handle socket creation ourselves

Yeah, that's a shortcoming in paho-mqtt at the moment. Alternatively, we can add a non-blocking way to create the socket in paho-mqtt. My only gripe about that approach is that the paho maintainers do not seem so interested in my pull requests. Maybe others have more luck.

Of course I fantasised about writing my own client (possibly using the Paho C library to handle the protocol, or just implementing the MQTT v5 myself), but it's not really a project I can dedicate enough time to.

Sounds like a fun project. :) I can relate to the time aspect of it, though. 😅

[agronholm]

I, too, found this in search of an AnyIO based MQTT client. I took a look at several alternatives, and asyncio-mqtt had by far the best looking API. The ideal client library would be one with this API but with asynchronous internal machinery that uses AnyIO's APIs. I'm surprised nobody has create a sans-io implementation of MQTT (any version) for Python. That would be even better!

I would do this work myself, but I have my hands full at the moment busy with AnyIO itself, APScheduler and other projects that have not had the attention they deserve. Perhaps something like smtpproto could serve as a basis? The MQTT protocol, even v5, isn't that complicated.

If I do this, would you relinquish the anyio-mqtt name @flyte ?

[norwood867]

agronholm, i'm curious about what you needed up using for a concurrent mqtt client?

[agronholm]

What do you mean by "concurrent" mqtt client?

[flyte]

If I do this, would you relinquish the anyio-mqtt name @flyte ?

Apologies for not seeing this sooner.

I'm happy to relinquish the name for a good cause! I'm not actively developing it now anyway due to having other mouths to feed with code :)

[norwood867]

What do you mean by "concurrent" mqtt client?

mqtt library that utilizes the async concurrency library like Trio or AnyIO. Asyncio-mqtt is solid from what I can tell, but my stuff is built on the trio, and my hacked-together mqtt access could be more robust.

[agronholm]

What do you mean by "concurrent" mqtt client?

mqtt library that utilizes the async concurrency library like Trio or AnyIO. Asyncio-mqtt is solid from what I can tell, but my stuff is built on the trio, and my hacked-together mqtt access could be more robust.

Concurrency is not the point, but asynchronous operation is. The rest of the app is async, and the async programming model has the benefit of thread safety (by virtue of using a single thread for everything) and the ability to cancel operations, which blocking sockets don't have. Additionally, any calls to blocking functions need to be wrapped in threads, using asyncio.to_thread() or similar. This can be problematic in high traffic applications that often require threads for other things and which can exhaust the number of threads. This unfortunately happens frequently in our app.

[frederikaalund]

I have a feeling that we'll see an anyio-like interface within asyncio in the standard library in a future Python release. It's not everything that anyio is (maybe we'll never get all of it in the standard library) but it's certainly a great move towards structured concurrency.

[agronholm]

I discussed this last week with Guido van Rossum, who is just about the only asyncio maintainer left, and the consensus was that asyncio won't get any esoteric new features anymore, and people should perhaps just migrate to something else, like Trio. Of course, given that Trio's compatibility with asyncio stuff is a bit iffy (trio-asyncio has some jagged edges), AnyIO offers more comfortable middle ground.