README.md

Using Exemplars in OpenTelemetry .NET

Exemplars are example data points for aggregated data. They provide specific context to otherwise general aggregations. One common use case is to gain ability to correlate metrics to traces (and logs). While OpenTelemetry .NET supports Exemplars, it is only useful if the telemetry backend also supports the capabilities. This tutorial uses well known open source backends to demonstrate the concept. The following are the components involved:

• Test App - We use existing example app from the repo. This app is already instrumented with OpenTelemetry for logs, metrics and traces, and is configured to export them to the configured OTLP end point.
• OpenTelemetry Collector - An instance of collector is run, which receives telemetry from the above app using OTLP. The collector then exports metrics to Prometheus, traces to Tempo.
• Prometheus - Prometheus is used as the Metric backend.
• Tempo - Tempo is used as the Tracing backend.
• Grafana - UI to query metrics from Prometheus, traces from Tempo, and to navigate between metrics and traces using Exemplar.

All these components except the test app require additional configuration to enable Exemplar feature. To make it easy for users, these components are pre-configured to enable Exemplars, and a docker-compose is provided to spun them all up, in the required configurations.

Pre-requisite

Install docker: https://docs.docker.com/get-docker/

Setup

As mentioned in the intro, this tutorial uses OTel Collector, Prometheus, Tempo, and Grafana, and they must be up and running before proceeding. The following spins all of them with the correct configurations to support Exemplars.

Navigate to current directory and run the following:

docker-compose up -d

If the above step succeeds, all dependencies would be spun up and ready now. To test, navigate to Grafana running at: http://localhost:3000/.

Run test app

Now that the required dependencies are ready, lets run the demo app. This tutorial is using the existing ASP.NET Core app from the repo.

Navigate to Example Asp.Net Core App directory and run the following command:

dotnet run

Once the application is running, navigate to http://localhost:5000/weatherforecast from a web browser. You may use the following Powershell script to generate load to the application.

while($true)
{
    Invoke-WebRequest http://localhost:5000/weatherforecast
    Start-Sleep -Milliseconds 500
}

Use Exemplars to navigate from Metrics to Traces

The application sends metrics (with exemplars), and traces to the OTel Collector, which export metrics and traces to Prometheus and Tempo respectively.

Please wait for 2 minutes before continuing so that enough data is generated and exported.

Open Grafana, select Explore, and select Prometheus as the source. Select the metric named "http_server_duration_bucket", and plot the chart. Toggle on the "Exemplar" option from the UI and hit refresh.

The Exemplars appear as special "diamond shaped dots" along with the metric charts in the UI. Select any Exemplar to see the exemplar data, which includes the timestamp when the measurement was recorded, the raw value, and trace context when the recording was done. The "trace_id" enables jumping to the tracing backed (tempo). Click on the "Query with Tempo" button next to the "trace_id" field to open the corresponding Trace in Tempo.

References

• Exemplar specification
• Exemplars in Prometheus
• Exemplars in Grafana
• Tempo