Telemetry: publish events for each actor create / restart / terminate

[Aaronontheweb]

Is your feature request related to a problem? Please describe.

I've been thinking about this in a few areas:

1. Phobos - right now we use invocation counters to keep track of actor starts / stops to keep track the churn of actors. In platforms like Prometheus we compute the delta between start / stop events to compute the total number of alive actors. We can try using things like an OTel UpDownCounter (not yet implemented in .NET) when that gets added to reach a total number and maybe that will be more accurate.
2. Petabridge.Cmd - similar to how we report on shard counts right now, I'd love to update the actor commands to show a breakdown of how many actors are alive by hierarchy position and type.
3. Homespun telemetry solutions - right now most of these are done via the logging system or subscribing to various data channels inside Akka.NET (i.e. Cluster.Subscribe), but the actual structure of the actor hierarchy itself is difficult to capture here.

In particular, some of the work I've been doing around Phobos 2.0 has been leading me to wonder if it makes sense to emit some basic telemetry around actor creation inside Akka.NET itself.

Describe the solution you'd like
I'll have to flesh this out in some more detail, but basically add three EventStream events that can be subscribed to:

public readonly struct ActorStarted{
   public IActorRef Ref {get;} // can be used for topological sort / correlation
   public Type ActorType {get;}
}

public readonly struct ActorRestarted{
   public IActorRef Ref {get;} // can be used for topological sort / correlation
   public Type ActorType {get;}
   public Exception CrashReason {get;}
}

public readonly struct ActorTerminated{
   public IActorRef Ref {get;} // can be used for topological sort / correlation
   public Type ActorType {get;}
   public TerminatedReason Reason {get;} // stopped, parent decided, system shutting down
}

Having a TerminatedReason would be really helpful - implementing that alone would be a bit of work, but I think it's achievable.

[Zetanova]

View points to add from my side:

1. There should be some gate to opt-in for this event creation
   something like _system.Diagnostic.TraceActorRestartEnabled = true
2. If you create a "Reason" class, please make it sealed and abstract to be used for all errors/execption
   I am using currently a private Reason class and it works fine over boundaries (remoting).
   It contains simply:

   public sealed class Reason
   {   
          public Guid Error { get; }
          
          public string Type { get;  }
   
          public string Message { get;  }
   }

   static singletons can be used for default reasons like 'Canceled', 'Timeout' ...
   and a ErrorService (extension) to tag Exception and convert them to a Reason

public readonly struct ActorTerminated
{
   public IActorRef Ref {get;} // can be used for topological sort / correlation
   public Type ActorType {get;}
   public Reason Reason {get;} // stopped, parent decided, system shutting down
}

//demo:
var t1 = new ActorTerminated(Self, GetType(), DefaultTerminationReasons.Stopped);

var reason = Context.System.GetExtensions<ErrorService>().ToReason(ex);
var t2 = new ActorTerminated(Self, GetType(), reason);

[Aaronontheweb]

_system.Diagnostic.TraceActorRestartEnabled = true

I was thinking this as well - since the events aren't intended to go over the network (due to the Type serialization) I was thinking of making it opt-out, but maybe I could stringify the actor type so that's not an issue.

[jason-field]

Having this would be also beneficial for actors managed by a BackoffSupervisor, where the child actor being "managed" must be initialised by the parent as part of its startup (Think FSM). How would the actor processing the event be able to distinguish between any actor's events and only its child hierarchy events?

[Aaronontheweb]

That would be up to whatever consumes the telemetry I think - if you see a bunch of actor create / terminate statements all stemming from the same BackoffSupervisor that would tell you that its children are having a lot of trouble staying alive, i.e. Akka.Persistence might be struggling to keep up.

[Aaronontheweb]

I have drafted an initial implementation of this concept here: #6294