Persistence and async/await

[AlexeyRokhin]

I haven't found Persist methods with Task returning handlers. All Persist... methods accept a simple Action<TEvent> delegate.

• Is there any architecture restriction that prevented the implementation of Persist methods which accept Func<TEvent, Task> delegate?
• Is there any workaround to use async handlers?

[Danthar]

Ignoring any possible architecture issues for a moment. What is your use-case for this requirement ?

[AlexeyRokhin]

I'd like to use Persist and async event handlers which use await for invocation of external services.

[Danthar]

So what you would really want is a ReceiveCommandAsync then. Combined with an PersistAsync .
Other contributors might disagree with me. But to be calling external services in your persistent actor feels like the wrong place, design wise.

[AlexeyRokhin]

calling external services in your persistent actor feels like the wrong place...

Why?

We are using microservices and we don't want to put all the logic into a single service with actors. We want to use actors as stateful objects with changing behavior (kind of FSM) but a significant part of required functionality (payments etc.) will be (and are) implemented in other microservices which don't use actors. So, the actors are going to use those microservices.

Of course, we can use synchronous event handlers with PipeTo attached to the asynchronous invocations of those microservices or even use fire-and-forget pattern to send requests to those microservices (via message bus), then receive replies as separate message bus messages and route them to the actors but PipeTo slightly complicates the code whille fire-and-forget significantly complicates the design (however, has some advantages).

BTW, is there any actor-related design patterns for such scenarios (and for other ones)?

[Danthar]

Well i guess it depends on what you need to achieve.

All in all i think it boils down to the basic actor strategies of "Let it Crash" meaning. Delegate dangerous work to child actors, and arrange their error handling with whatever supervision strategy you need.
How i would model this is by having 1 actor acts as the orchestrator that receives your commands, then uses child actors to collect the extra data required leveraging supervision strategies to handle errors. And then communicate the end result to your persistent actor.

You might also have your persistent actor be the orchistrator. But personally i like the separation of concern here.

[Aaronontheweb]

Hi @AlexeyRokhin

So what you're asking for is an overload of the Persist methods that returns a Task - that way you can do async side effects after the event has been persisted. Am I understanding you correctly?

[Danthar]

What CommandAsync api with regards to the PersistentActor abstract base class(es) are you referring to ?

The base classes for PersistentActors are a bit of a maze. But i think supporting Task PersistAsync besides a normal void Persist would pose some interesting challenges. Especially if you want to support mixing the 2 in a single PersistentActor implementation.

Additionally, considering how those callbacks in the persist function are handled internally, with all the lifecycle stuff, i might be missing some trick we can use, but it would have some pretty impactful knock on effects in the internal API it looks like.

[Danthar]

To be clear though. Im not against this feature/change. I understand the appeal. Its just not how i would design my persistent actors. And looking at the internals it would probably be quite abit of effort.

[to11mtm]

Would something like a public void Persist<TEvent>(TEvent @event, Func<TEvent,Task> asyncHandler) overload potentially be less of a plumbing headache to implement?

[AlexeyRokhin]

So what you're asking for is an overload of the Persist methods that returns a Task - that way you can do async side effects after the event has been persisted.

@Aaronontheweb yes. That's correct.

BTW, you are talking about side effects: so, the expected way to use persistence is to perform the required message processing before persisting events?
What if I need to persist event first and then process the message? For instance, in case when a cluster node that hosts a specific actor fails after the event persisted and before/during the message processed, I'd ike to resume message processing after the actor recovers (let's say that message processing is idempotent).

[Danthar]

@to11mtm Nope. because thats exactly the problem.
Persistent actors do batching internally. Meaning updates get batched before the go to the journal.
Calling Persist or any of its overloads is not a blocking operation. What it does it adds the writing of the event into a batch, and enqueue's the callback.
Then later on when the batch is processed, it dequeu's all pending callbacks and goes through them one by one.

But by that time, your no longer in the context of the original Command handler that 'scheduled' the update.

I took a longer time to look into this and found the CommandAsync in the ReceivePersistentActor that @Aaronontheweb was referring to.
Unfortunately thats not going to help us because of the aforementioned. (the callback handler is never executed at that point)

If we want to support this, then we would have to do some hefty refactoring in the underlying EventSourced class in order to support processing persistent actor states in an async nature, with the goal of being able to await the enqueue'd callbacks that are registered with the Persist calls. See Eventsourced.Lifecycle for more details. Also it seems like it hooks into AroundReceive to process messages, which is another snag.

The reason why the ReceiveActor infrastructure can support async await is because its practically a layer on top of the baseActor stuff, and has its own RunTask implementation to allow the Receive handler to run in a blocking manor on the actor. When you drop a level, such is the underlying Eventsourced persistentactor stuff is doing, you lose that.

We might be able to introduce a layer here to allow us to run things in a async nature, such as the ReceiveActor infrastructure allows.
But it would mean enforcing this environment for the entire Eventsources.Lifecycle stuff. Which would probably impact performance considering we'd be spinning up Tasks all the time, even for actions which will not be doing any async work.

All this is assuming that simply invoking the handler and ignoring the task that comes out of it is unacceptable.
Which it is in my mind. Because even if we register a continueWith to handle any errors, i see no way to communicate this error to the actor in the proper way in the current design.