diff --git a/docs/sources/tempo/configuration/_index.md b/docs/sources/tempo/configuration/_index.md index 1273857f5ee..b1918cda847 100644 --- a/docs/sources/tempo/configuration/_index.md +++ b/docs/sources/tempo/configuration/_index.md @@ -203,7 +203,7 @@ distributor: [enabled: | default = false] [include_all_attributes: | default = false] [filter_by_status_error: | default = false] - + # Optional. # Enable to log every discarded span to help debug ingestion or calculate span error distributions using the logs. log_discarded_spans: @@ -1559,6 +1559,10 @@ overrides: # Per-user compaction window. If this value is set to 0 (default), # then block_retention in the compactor configuration is used. [compaction_window: | default = 0s] + # Allow compaction to be deactivated on a per-tenant basis. Default value + # is false (compaction active). Useful to perform operations on the backend + # that require compaction to be disabled for a period of time. + [compaction_disabled: | default = false] # Metrics-generator related overrides metrics_generator: diff --git a/docs/sources/tempo/metrics-generator/_index.md b/docs/sources/tempo/metrics-generator/_index.md index 632232e097a..9585fd3510e 100644 --- a/docs/sources/tempo/metrics-generator/_index.md +++ b/docs/sources/tempo/metrics-generator/_index.md @@ -68,8 +68,11 @@ When multi-tenancy is enabled, the metrics-generator forwards the `X-Scope-OrgID ## Native histograms -The metrics-generator supports the ability to produce [native histograms](https://grafana.com/docs/grafana-cloud/whats-new/native-histograms/), for -high-resolution data. Users must [update the receiving endpoint](https://grafana.com/docs/mimir/latest/configure/configure-native-histograms-ingestion/) to ingest native -histograms, and [update histogram queries](https://grafana.com/docs/mimir/latest/visualize/native-histograms/) in their dashboards. +[Native histograms](https://grafana.com/docs/grafana-cloud/whats-new/native-histograms/) are a data type in Prometheus that can produce, store, and query high-resolution histograms of observations. +It usually offers higher resolution and more straightforward instrumentation than classic histograms. + +The metrics-generator supports the ability to produce native histograms for +high-resolution data. Users must [update the receiving endpoint](https://grafana.com/docs/mimir//configure/configure-native-histograms-ingestion/) to ingest native +histograms, and [update histogram queries](https://grafana.com/docs/mimir//visualize/native-histograms/) in their dashboards. To learn more about the configuration, refer to the [Metrics-generator]({{< relref "../configuration#metrics-generator" >}}) section of the Tempo Configuration documentation. diff --git a/docs/sources/tempo/traceql/_index.md b/docs/sources/tempo/traceql/_index.md index 55e07d0815f..6dd929674af 100644 --- a/docs/sources/tempo/traceql/_index.md +++ b/docs/sources/tempo/traceql/_index.md @@ -64,17 +64,41 @@ In this example, the search reduces traces to those spans where: Queries select sets of spans and filter them through a pipeline of aggregators and conditions. If, for a given trace, this pipeline produces a spanset then it is included in the results of the query. - ## Selecting spans -TraceQL differentiates between two types of span data: intrinsics, which are fundamental to spans, and attributes, which are customizable key-value pairs. You can use intrinsics and attributes to build filters and select spans. - In TraceQL, curly brackets `{}` always select a set of spans from the current trace. They are commonly paired with a condition to reduce the spans being passed in. -### Intrinsic fields +TraceQL differentiates between two types of span data: intrinsics, which are fundamental to spans, and attributes, which are customizable key-value pairs. +You can use intrinsics and attributes to build filters and select spans. + +Intrinsic fields are fundamental to scopes. +Intrinsics are inherently present, as opposed to other key-value pairs (attributes) that are added by a developer. + +Intrinsics are always indicated using a `:`. +Refer to the Intrinsics table for all current intrinsics. + +Custom attributes are prefixed with . such as `span.`, `resource.` , `link.`, or `event`. +Resource has no intrinsic values. +It only has custom attributes. +The `trace` scope is only an intrinsic and doesn't have any custom attributes at the trace level. + +Intrinsics example: +``` +{ span:name = "foo" } +{ event:name = "foo" } +{ trace:id = "1234" } +{ link:traceID = "1234" } +``` + +Attributes example: +``` +{ span.foo = "bar" } +{ resource.foo = "bar" } +{ link.foo = "bar" } +{ event.foo = "bar" } +``` -Intrinsic fields are fundamental to spans. These fields can be referenced when selecting spans. -Custom attributes are prefixed with `.`, `span.` or `resource.`, whereas intrinsics are typed directly. +### Intrinsic fields The following table shows the current available scoped intrinsic fields: @@ -94,10 +118,13 @@ The following table shows the current available scoped intrinsic fields: | `event:timeSinceStart` | duration | time of event in relation to the span start time | `{ event:timeSinceStart > 2ms}` | | `link:spanID` | string | link span id using hex string | `{ link:spanID = "0000000000000001" }` | | `link:traceID` | string | link trace id using hex string | `{ link:traceID = "1234567890abcde" }` | + + -`trace:duration`, `trace:rootName`, and `trace:rootService` are trace-level intrinsics and are the same for all spans in the same trace. +The trace-level intrinsics, `trace:duration`, `trace:rootName`, and `trace:rootService`, are the same for all spans in the same trace. Additionally, these intrinsics are significantly more performant because they have to inspect much less data then a span-level intrinsic. They should be preferred whenever possible to span-level intrinsics. @@ -111,15 +138,24 @@ This example searches all Kubernetes clusters called `service-name` that have a { resource.k8s.cluster.name="service-name" && trace:rootName !~ ".*perf.*"} ``` + ### Attribute fields -TraceQL has five different attribute scopes: span attributes, resource attributes, event attributes, link attributes, instrumentation scope attributes. By expanding a span in the Grafana UI, you can see both its span attributes (1 in the screenshot) and resource attributes (2 in the screenshot). +TraceQL has four different attribute scopes: span attributes, resource attributes, event attributes, and link attributes. + + +By expanding a span in the Grafana UI, you can see both its span attributes (1 in the screenshot) and resource attributes (2 in the screenshot).

Example of span and resource attributes.

-Attribute fields are derived from the span and can be customized. Process and span attribute types are [defined by the attribute itself](https://github.com/open-telemetry/opentelemetry-proto/blob/b43e9b18b76abf3ee040164b55b9c355217151f3/opentelemetry/proto/common/v1/common.proto#L30-L38), whereas intrinsic fields have a built-in type. You can refer to dynamic attributes (also known as tags) on the span or the span's resource. +Attribute fields are derived from the span and can be customized. +Process and span attribute types are [defined by the attribute itself](https://github.com/open-telemetry/opentelemetry-proto/blob/b43e9b18b76abf3ee040164b55b9c355217151f3/opentelemetry/proto/common/v1/common.proto#L30-L38), whereas intrinsic fields have a built-in type. +You can refer to dynamic attributes (also known as tags) on the span or the span's resource. -Attributes in a query start with a span scope (for example, `span.http`) or resource scope (for example, `resource.namespace`) depending on what you want to query. This provides significant performance benefits because it allows Tempo to only scan the data you are interested in. +Attributes in a query start with a span, resource, event, or link scope. +For example, you could use `span.http` or `resource.namespace`, depending on what you want to query. +This provides significant performance benefits because it allows Tempo to only scan the data you are interested in. To find traces with the `GET HTTP` method, your query could look like this: @@ -128,6 +164,7 @@ To find traces with the `GET HTTP` method, your query could look like this: ``` For more information about attributes and resources, refer to the [OpenTelemetry Resource SDK](https://opentelemetry.io/docs/reference/specification/resource/sdk/). + #### Examples Find traces that passed through the `production` environment: @@ -140,20 +177,29 @@ Find any database connection string that goes to a Postgres or MySQL database: { span.db.system =~ "postgresql|mysql" } ``` +You can use the `event` scope to query events that happen within a span. +A span event is a unique point in time during the span’s duration. +While spans help build the structural hierarchy of your services, span events can provide a deeper level of granularity to help debug your application faster and maintain optimal performance. +To learn more about how you can use span events, read the [What are span events?](https://grafana.com/blog/2024/08/15/all-about-span-events-what-they-are-and-how-to-query-them/) blog post. + You can query for an exception in your span event: ``` { event.exception.message =~ ".*something went wrong.*" } ``` +If you've instrumented your traces for span links, you can use the `link` scope to query the link data. A span link associates one span with one or more other spans that are a casual relationship. +For more information on span links, refer to the [Span Links](https://opentelemetry.io/docs/concepts/signals/traces/#span-links) documentation in the Open Telemetry project. + You can search for an attribute in your link: ``` { link.opentracing.ref_type = "child_of" } ``` - + ### Unscoped attribute fields