feat: TraceQL metrics: average over time

docs/sources/tempo/traceql/metrics-queries/_index.md

@@ -60,7 +60,7 @@ Refer to [Solve problems using metrics queries](./solve-problems-metrics-queries
 
 ### Functions
 
-TraceQL metrics queries currently include the following functions for aggregating over groups of spans: `rate`, `count_over_time`, `quantile_over_time`, `histogram_over_time`, and `compare`.
+TraceQL metrics queries currently include the following functions for aggregating over groups of spans: `rate`, `count_over_time`, `max_over_time`, `min_over_time`, `avg_over_time`, `quantile_over_time`, `histogram_over_time`, and `compare`.
 These functions can be added as an operator at the end of any TraceQL query.
 
 For detailed information and example queries for each function, refer to [TraceQL metrics functions](./functions).

docs/sources/tempo/traceql/metrics-queries/functions.md

@@ -12,7 +12,7 @@ keywords:
 
 <!-- If you add a new function to this page, make sure you also add it to the _index.md#functions section.-->
 
-TraceQL supports `rate`, `count_over_time`, `quantile_over_time`, `histogram_over_time`, and `compare` functions.
+TraceQL supports `rate`, `count_over_time`, `min_over_time`, `avg_over_time`, `quantile_over_time`, `histogram_over_time`, and `compare` functions.
 
 ## Available functions
 
@@ -30,6 +30,9 @@ These functions can be added as an operator at the end of any TraceQL query.
 `max_over_time`
 : Returns the minimum value for the specified attribute across all matching spans per time interval (refer to the [`step` API parameter](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/#traceql-metrics)).
 
+`avg_over_time`
+: Returns the average value for the specified attribute across all matching spans per time interval (refer to the [`step` API parameter](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/#traceql-metrics)).
+
 `quantile_over_time`
 : The quantile of the values in the specified interval
 
@@ -94,7 +97,7 @@ This example counts the number of spans with name `"GET /:endpoint"` broken down
 
 ```
 
-## The `min_over_time` and `max_over_time` functions
+## The `min_over_time`, `max_over_time`, and `avg_over_time`  functions
 
 The `min_over_time()` function lets you aggregate numerical attributes by calculating their minimum value.
 For example, you could choose to calculate the minimum duration of a group of spans, or you could choose to calculate the minimum value of a custom attribute you've attached to your spans, like `span.shopping.cart.entries`.
@@ -103,11 +106,14 @@ The time interval that the minimum is computed over is set by the `step` paramet
 The `max_over_time()` let you aggregate numerical values by computing the maximum value of them, such as the all important span duration.
 The time interval that the maximum is computer over is set by the `step` parameter.
 
+The `avg_over_time()` function lets you aggregate numerical values by computing the maximum value of them, such as the all important span duration.
+The time interval that the maximum is computer over is set by the `step` parameter.
+
 For more information, refer to the [`step` API parameter](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/#traceql-metrics).
 
 ### Parameters
 
-Numerical field that you want to calculate the minimum or maximum of.
+Numerical field that you want to calculate the minimum, maximum, or average of.
 
 ### Examples
 
@@ -134,6 +140,16 @@ This example computes the maximum duration for each `http.target` of all spans n
 { name = "GET /:endpoint" } | max_over_time(span.http.response.size)
 ```
 
+This example computes the average duration for each `http.status_code` of all spans named `"GET /:endpoint"`.
+
+```
+{ name = "GET /:endpoint" } | avg_over_time(duration) by (span.http.status_code)
+```
+
+```
+{ name = "GET /:endpoint" } | avg_over_time(span.http.response.size)
+```
+
 ## The `quantile_over_time` and `histogram_over_time` functions
 
 The `quantile_over_time()` and `histogram_over_time()` functions let you aggregate numerical values, such as the all important span duration.

pkg/traceql/engine_metrics.go

@@ -17,6 +17,8 @@ import (
 )
 
 const (
+	internalLabelMetaType = "__meta_type"
+	internalMetaTypeCount = "__count"
 	internalLabelBucket   = "__bucket"
 	maxExemplars          = 100
 	maxExemplarsPerBucket = 2
@@ -176,10 +178,12 @@ func (ls Labels) String() string {
 			promValue = "<nil>"
 		case l.Value.Type == TypeString:
 			s := l.Value.EncodeToString(false)
-			if s != "" {
-				promValue = s
-			} else {
+			if s == "nil" {
+				promValue = "<nil>"
+			} else if s == "" {
 				promValue = "<empty>"
+			} else {
+				promValue = s
 			}
 		default:
 			promValue = l.Value.EncodeToString(false)