Add hooks so that metrics tools can generate metrics on key events

[pjfanning]

One of the reasons to open this discussion is that there are a few ways to do this and it would be good to debate some of the approaches. Ideally, we would have a solution that has low performance head and close to zero overhead if no metrics tools are in use.

#1484 will probably be implemented in Pekko 1.1.2 to get the Kamon metrics tool to work with Pekko again. It is likely preferable in future Pekko versions to have an endorsed way to gather the metrics so that we can avoid breaking the metrics tools when we make changes to the Pekko internals. Kamon is basically instrumenting the Pekko code at runtime and is depending on some private functions and if these change name, param signature or get inlined, the Kamon instrumentation will stop working.

To summarise the Pekko 1.1 situation.

• Pekko 1.1 build enabled Scala 2 compiler inlining
• This means that functions marked as final or functions in classes marked as final - or functions explicitly annotated with @inline annotation can be inlined by the Scala compiler
• This broke some Kamon and micrometer-pekko functionality that relied on AOP or similar approaches to instrument named functions in Pekko code.
• The aim for Pekko 1.1.2 is to add some @noinline annotations to force some functions not to be inlined. Nothing was inlined in Pekko 1.0. If you find @noinline annotations in Pekko code, this is likely to be the reason.
• Hopefully, we will find a better way to support metrics introspection in Pekko 1.2/2.0 - the point of this discussion.

I've raised #1499 to discuss Context Propagation.

[Roiocam]

AFAIK, there are ways to achieve observability:

• intrusive ways: like kakfa(or pekko projection), we expose the interface of the key method(as you said hook), and let the users choose the meter library.
• partial non-intrusive ways: keep most things unchanged, add the particular method for kamon.
• fully non-intrusive ways: keep things unchanged, using instrumentation library to achieve this (like kamon, opentelemetry)

for me, i think the observability should be non-intrusive, the key method signature barely won't change, the only thing we need to care about is inline.

[pjfanning]

I sort of like the idea of using the Pekko Event Stream or something similar to avoid the need for manipulating the Pekko code at runtime like Kamon does today. If we thought that the using the existing Event Stream would overload the existing stream, we could add a specific event stream for observability.

One existing but very specific Pekko metrics support for clusters - https://pekko.apache.org/docs/pekko/1.1/cluster-metrics.html

As well as allowing observability libs to register interest in events, we may also need to allow pluggability to allow these observability libs to control how much is written to the events. Many existing events would need to remain as is because the events are needed in their existing format for existing purposes - but it might be useful to allow more information to be added to some events or for additional events to be added. The existing info that can be gleaned from the existing events may not be enough for the observability libs.