Options for Micrometer in Helidon

Intro

What is Micrometer

Micrometer [1] is

a simple facade over the instrumentation clients for the most popular monitoring systems

with the aim of insulating application developers from particular metrics implementations and their APIs.

Micrometer elsewhere (very briefly)

MicroProfile

Recent discussions on the MicroProfile mailing list suggest that in an upcoming major release (likely MP Metrics 4.0), MP will adopt Micrometer as the (preferred) API. There has been conversation about perhaps dropping JSON output support. In the Micrometer world, different output formats take the form of different MeterRegistry instances which is a bit different from the MP approach and our implementation in which the knowledge of how to format a metric is embedded in each metric's implementation class.

Quarkus

Quarkus supports MicroProfile and Micrometer but prefers the Micrometer API, and it provides a Quarkus extension which maps the MP metrics API to Micrometer.

Terminology

Meters and metrics

A Micrometer meter refers to what MP refers to as a metric: the prescription to collect data of a given type (counter, timing, etc.). Micrometer identifies each meter with a name and zero or more tags (as with MP metrics).

In Micrometer, a metric is a single data observation of a meter.

Registry

In Micrometer, a registry is a collection of meters (and their metrics) to be output in the same, particular way. Micrometer's web page currently lists 18 different implementations (including Prometheus). Differences in output among various registry implementations can be push vs. pull as well as different formats of output. Individual meters are registered with a registry.

Micrometer provides a CompositeRegistry which is a collection of registries and meters. Adding a meter to a composite registry adds it to all the collected registries, and a registry added to a composite registry includes all the meters previously registered with the composite registry.

For example, here is one way to approach this for something like Helidon:

• the app (or a framework) could use and expose to developers a CompositeRegistry
• Helidon (through configuration) or an app (via a hypothetical Helidon MicrometerSupport class) could add specific registries for specific formats or output techniques
• the app registers meters with the exposed CompositeRegistry
• when metrics are pulled, choose which specific registry's format is requested and get the output from that registry for return to the client.

By contrast, in Helidon metrics today a registry typically contains a given category of metrics and there are three built-in ones: base, vendor, and application. Each metric implementation knows how to express itself in both Prometheus and JSON formats.

Main types of meters exposed

• counter - monotonically increasing value
• gauge - current value of some varying quantity that typically has an upper bound
• timer - total time and count of a short-duration activity
• distribution summary - tracks distribution of values over a range (somewhat similar to MP histogram)
• long-task timer - timer with added functionality for long-running actions (e.g., while the action is in progress, the value is available and alerts can be delivered at reporting intervals)

Micrometer provides various specialized variants of some of these with added behavior.

Simplest example: adding Micrometer/Prometheus to an SE app (no changes to Helidon)

Using the Helidon SE quickstart:

1. Add the Micrometer dependency:

   <dependency>
       <groupId>io.micrometer</groupId>
       <artifactId>micrometer-registry-prometheus</artifactId>
       <version>1.6.1</version>
   </dependency>

2. Create an instance of PrometheusMeterRegistry during server start-up and register an endpoint at /micrometer: Main#createRouting

       private static Routing createRouting(Config config) {
           MetricsSupport metrics = MetricsSupport.create();
           prometheusMeterRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT); // [1]
           GreetService greetService = new GreetService(config, prometheusMeterRegistry);   // [2]
           HealthSupport health = HealthSupport.builder()
                   .addLiveness(HealthChecks.healthChecks())   // Adds a convenient set of checks
                   .build();
   
           return Routing.builder()
                   .register(health)                   // Health at "/health"
                   .register(metrics)                  // Metrics at "/metrics"
                   .get("/micrometer", (req, resp) -> resp.send(prometheusMeterRegistry.scrape())) // [3]
                   .register("/greet", greetService)
                   .build();
       }

   [1] Creates and saves the Prometheus Micrometer registry instance.
   [2] Passes the registry to the service so it can create a meter (a counter of all get invocations).
   [3] Sets up the /micrometer endpoint which reports the contents of the Micrometer registry.
3. Change GreetService.java:
   1. Change constructor:

          GreetService(Config config, MeterRegistry meterRegistry) {
              this.meterRegistry = meterRegistry;                          // [1]
              getCounter = meterRegistry.counter("greeting.get");          // [2]
              greeting.set(config.get("app.greeting").asString().orElse("Ciao"));
          }

   2. Change routing:

          public void update(Routing.Rules rules) {
              rules
                  .get((ServerRequest req, ServerResponse resp) -> { // [3]
                      getCounter.increment();
                      req.next();
                  })
                  .get("/", this::getDefaultMessageHandler)
                  .get("/{name}", this::getMessageHandler)
                  .put("/greeting", this::updateGreetingHandler);
          }

   [1] Saves the meter registry.
   [2] Creates the counter in the meter registry to count gets.
   [3] Adds a handler for all gets which updates the get counter.

Proposed strategy for Micrometer and Helidon SE and MP

Immediate term

Write a blog article showing and describing the above example, illustrating how to use Micrometer with Helidon today.

Short-term

1. Provide a simple io.helidon.metrics.micrometer:helidon-micrometer module containing a MicrometerSupport class to make it even easier than the example above for developers to use Micrometer from their Helidon apps.

   This would not change any of the existing functionality around metrics in Helidon. It would purely be a way developers could add Micrometer support to their SE or MP app. There is one Micrometer annotation -- @Timed -- which we could support in a Helidon MP Micrometer module.

   Key features of Helidon Micrometer support (with some minor issues we'd need to resolve):

   1. /micrometer endpoint (changeable by configuration)

   2. Provide easy support (perhaps by default) for the Prometheus Micrometer registry, configurable from Helidon config.

   3. Possibly add our own registry that provides JSON output. (Quarkus has this feature for their Micrometer support.)

   4. Allow the app to enroll Micrometer registries of its choice with MicrometerSupport.

      This is a little more complicated than one might expect. There is no abstract method defined by the abstract class MeterRegistry (which other registries extend) for producing output; each implementation has its own way. Further, we need to be able to select which MeterRegistry to use in response to a given request to the endpoint. In Helidon metrics today, we use the request's accepted media type: text/plain means Prometheus, application/json means JSON.

      A possible solution for both: an app which enrolls a MeterRegistry instance with our MicrometerSupport object must provide an associated Function<ServerRequest, Optional<Handler>> which:

      • checks if the associated MeterRegistry instance should be used in responding to the ServerRequest; (it can look at the media type, query parameters, etc.)
      • if so, returns an Optional<Handler> of an actual handler that retrieves the data from the associated meter registry and uses it to set the response's status and entity;
      • if not, return Optional.empty() to indicate that the associated meter registry is not suitable for this particular request.

   5. Allow the app to add its own meters and update their values, both programmatically in the app code. MicrometerSupport itself might expose many of the MeterRegistry methods so the app could use it directly to register metrics in all the registries enrolled.

2. Add a short doc section to the web site.

3. Write a short blog about the new Micrometer support and how to use it.

Non-goals:

• replacing the MP-style metrics in SE and MP with Micrometer's equivalents and near-equivalents
• mapping the MP annotations to Micrometer
• inventing our own additional annotations for the non-Timer meter types

Medium-term

Investigate how the Quarkus extension maps MP metrics to Micrometer, in particular how they handle some of the misalignments that have been discussed in the MP Google group. Providing a similar adapter in Helidon could be useful to our developers and could be a good step for us toward eventual full migration to Micrometer in Helidon (if that's what MP decides to do).

This would build directly on the short-term work described above.

Long-term

The current MP metrics model permeates the Helidon SE and MP metrics implementation. Basically, except for annotations and interceptors, Helidon SE metrics is Helidon MP metrics.

There has been extensive discussion in the MP Google group [2] about MP metrics.next and Micrometer. It looks as if MP will adopt Micrometer as the preferred metrics API. Some existing MP metrics do not align perfectly with their Micrometer counterpart meters (e.g., timers, histograms) and those incompatibilities will need to be sorted out by MP.

Plan: Wait for the dust to settle with MP metrics.next, see how they plan to handle incompatibilities, what backward compatibility support they plan to offer, then do the same in our MP and SE implementations.

[1] https://micrometer.io/docs/concepts
[2] https://groups.google.com/g/microprofile/c/E-kWz47VDVg/m/acLqUxlEAgAJ