drilling down into metrics from traces

[jSwanson99]

Hi, I am working with open-source tempo and trying to figure out how I can take an input trace id, and get aggregate metrics about the spans that are present in the input trace. So for example: "I can see span A took 500ms in trace 123, whats its p50 latency?". I have prom metrics coming from the metrics-generator, so I know for this example I could just write a single prom query to see that. However, I am trying to figure out how to make a dashboard that takes a traceid as input and then does all of the work required to aggregate metrics on that trace's spans for you.

In short:

• Take a traceID as input
• Find all span names in the trace (how do I do this?)
• For each span name, run some prom queries to get me p50 latency for example (probably just promQL I figure)

I am mostly looking for suggestions on the second part of this, I figure part 3 is just some futzing with dashboard variables prom queries.

Thanks!

[joe-elliott]

I want this too, but it currently does not exist. There are two things missing. First, the trace:id intrinsic defined here:

https://github.com/grafana/tempo/blob/main/docs/design-proposals/2023-11%20TraceQL%20Extensions.md#scoped-intrinsics

There are a few complexities to implementing it but it shouldn't be too difficult.

Second, would be the ability to derive quantiles on duration from spans. The team is actively working on this and it should be available in the next release.

Once implemented you could run a metrics query that would look something like:

{ trace:id = "abcd" } | quantile(.5, duration) by (name)

[jSwanson99]

thanks for the answer, what you are describing is exactly what I am looking for!