Skip to content

Commit

Permalink
[release-v2.6] [DOC] TraceQL, metrics-gen, and overrides doc updates …
Browse files Browse the repository at this point in the history 
…for 2.6 (#4054)
  • Loading branch information
@knylander-grafana
knylander-grafana authored Sep 4, 2024
1 parent f46b5a3 commit 5db9717
Show file tree
Hide file tree
Showing 4 changed files with 99 additions and 32 deletions.
13 changes: 12 additions & 1 deletion docs/sources/tempo/configuration/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ The Tempo configuration options include:
- [Filter policy config](#filter-policy-config)
- [Filter policy](#filter-policy)
- [Policy match](#policy-match)
- [Examples](#examples)
- [Examples](#examples)
- [KVStore config](#kvstore-config)
- [Search config](#search-config)
- [WAL config](#wal-config)
Expand Down Expand Up @@ -204,6 +204,13 @@ distributor:
[include_all_attributes: <boolean> | default = false]
[filter_by_status_error: <boolean> | default = false]

# Optional.
# Enable to log every discarded span to help debug ingestion or calculate span error distributions using the logs.
log_discarded_spans:
[enabled: <boolean> | default = false]
[include_all_attributes: <boolean> | default = false]
[filter_by_status_error: <boolean> | default = false]

# Optional.
# Enable to metric every received span to help debug ingestion
# This is not recommended for production environments
Expand Down Expand Up @@ -1552,6 +1559,10 @@ overrides:
# Per-user compaction window. If this value is set to 0 (default),
# then block_retention in the compactor configuration is used.
[compaction_window: <duration> | default = 0s]
# Allow compaction to be deactivated on a per-tenant basis. Default value
# is false (compaction active). Useful to perform operations on the backend
# that require compaction to be disabled for a period of time.
[compaction_disabled: <bool> | default = false]
# Metrics-generator related overrides
metrics_generator:
Expand Down
9 changes: 6 additions & 3 deletions docs/sources/tempo/metrics-generator/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,11 @@ When multi-tenancy is enabled, the metrics-generator forwards the `X-Scope-OrgID

## Native histograms

The metrics-generator supports the ability to produce [native histograms](https://grafana.com/docs/grafana-cloud/whats-new/native-histograms/), for
high-resolution data. Users must [update the receiving endpoint](https://grafana.com/docs/mimir/latest/configure/configure-native-histograms-ingestion/) to ingest native
histograms, and [update histogram queries](https://grafana.com/docs/mimir/latest/visualize/native-histograms/) in their dashboards.
[Native histograms](https://grafana.com/docs/grafana-cloud/whats-new/native-histograms/) are a data type in Prometheus that can produce, store, and query high-resolution histograms of observations.
It usually offers higher resolution and more straightforward instrumentation than classic histograms.

The metrics-generator supports the ability to produce native histograms for
high-resolution data. Users must [update the receiving endpoint](https://grafana.com/docs/mimir/<MIMIR_VERSION>/configure/configure-native-histograms-ingestion/) to ingest native
histograms, and [update histogram queries](https://grafana.com/docs/mimir/<MIMIR_VERSION>/visualize/native-histograms/) in their dashboards.

To learn more about the configuration, refer to the [Metrics-generator]({{< relref "../configuration#metrics-generator" >}}) section of the Tempo Configuration documentation.
2 changes: 1 addition & 1 deletion docs/sources/tempo/setup/upgrade.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ The upgrade process changes for each version, depending upon the changes made fo

This upgrade guide applies to on-premise installations and not for Grafana Cloud.

For detailed information about any release, refer to the [Release notes](../release-notes/).
For detailed information about any release, refer to the [Release notes](https://grafana.com/docs/tempo/<TEMPO_VERSION>/setup/upgrade/).

{{< admonition type="tip" >}}
You can check your configuration options using the [`status` API endpoint]({{< relref "../api_docs#status" >}}) in your Tempo installation.
Expand Down
107 changes: 80 additions & 27 deletions docs/sources/tempo/traceql/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -64,38 +64,67 @@ In this example, the search reduces traces to those spans where:

Queries select sets of spans and filter them through a pipeline of aggregators and conditions. If, for a given trace, this pipeline produces a spanset then it is included in the results of the query.


## Selecting spans

TraceQL differentiates between two types of span data: intrinsics, which are fundamental to spans, and attributes, which are customizable key-value pairs. You can use intrinsics and attributes to build filters and select spans.

In TraceQL, curly brackets `{}` always select a set of spans from the current trace. They are commonly paired with a condition to reduce the spans being passed in.

### Intrinsic fields
TraceQL differentiates between two types of span data: intrinsics, which are fundamental to spans, and attributes, which are customizable key-value pairs.
You can use intrinsics and attributes to build filters and select spans.

Intrinsic fields are fundamental to scopes.
Intrinsics are inherently present, as opposed to other key-value pairs (attributes) that are added by a developer.

Intrinsics are always indicated using a `<scope>:`.
Refer to the Intrinsics table for all current intrinsics.

Custom attributes are prefixed with <scope>. such as `span.`, `resource.` , `link.`, or `event`.
Resource has no intrinsic values.
It only has custom attributes.
The `trace` scope is only an intrinsic and doesn't have any custom attributes at the trace level.

Intrinsics example:
```
{ span:name = "foo" }
{ event:name = "foo" }
{ trace:id = "1234" }
{ link:traceID = "1234" }
```

Attributes example:
```
{ span.foo = "bar" }
{ resource.foo = "bar" }
{ link.foo = "bar" }
{ event.foo = "bar" }
```

Intrinsic fields are fundamental to spans. These fields can be referenced when selecting spans.
Custom attributes are prefixed with `.`, `span.` or `resource.`, whereas intrinsics are typed directly.
### Intrinsic fields

The following table shows the current available scoped intrinsic fields:

| **Field** | **Type** | **Definition** | **Example** |
| ----------------------- | ----------- | --------------------------------------------------------------- | -------------------------------------- |
| `span:status` | status enum | status: error, ok, or unset | `{ span:status = ok }` |
| `span:statusMessage` | string | optional text accompanying the span status | `{ span:statusMessage = "Forbidden" }` |
| `span:duration` | duration | end - start time of the span | `{ span:duration > 100ms }` |
| `span:name` | string | operation or span name | `{ span:name = "HTTP POST" }` |
| `span:kind` | kind enum | kind: server, client, producer, consumer, internal, unspecified | `{ span:kind = server }` |
| `span:id` | string | span id using hex string | `{ span:id = "0000000000000001" }` |
| `trace:duration` | duration | max(end) - min(start) time of the spans in the trace | `{ trace:duration > 100ms }` |
| `trace:rootName` | string | if it exists the name of the root span in the trace | `{ trace:rootName = "HTTP GET" }` |
| `trace:rootService` | string | if it exists the service name of the root span in the trace | `{ trace:rootServiceName = "gateway" }`|
| `trace:id` | string | trace id using hex string | `{ trace:id = "1234567890abcde" }` |
| `event:name` | string | name of event | `{ event:name = "exception" }` |
| `event:timeSinceStart` | duration | time of event in relation to the span start time | `{ event:timeSinceStart > 2ms}` |
| `link:spanID` | string | link span id using hex string | `{ link:spanID = "0000000000000001" }` |
| `link:traceID` | string | link trace id using hex string | `{ link:traceID = "1234567890abcde" }` |

`trace:duration`, `trace:rootName`, and `trace:rootService` are trace-level intrinsics and are the same for all spans in the same trace.
| **Field** | **Type** | **Definition** | **Example** |
| ------------------------ | ----------- | --------------------------------------------------------------- | -------------------------------------- |
| `span:status` | status enum | status: error, ok, or unset | `{ span:status = ok }` |
| `span:statusMessage` | string | optional text accompanying the span status | `{ span:statusMessage = "Forbidden" }` |
| `span:duration` | duration | end - start time of the span | `{ span:duration > 100ms }` |
| `span:name` | string | operation or span name | `{ span:name = "HTTP POST" }` |
| `span:kind` | kind enum | kind: server, client, producer, consumer, internal, unspecified | `{ span:kind = server }` |
| `span:id` | string | span id using hex string | `{ span:id = "0000000000000001" }` |
| `trace:duration` | duration | max(end) - min(start) time of the spans in the trace | `{ trace:duration > 100ms }` |
| `trace:rootName` | string | if it exists the name of the root span in the trace | `{ trace:rootName = "HTTP GET" }` |
| `trace:rootService` | string | if it exists the service name of the root span in the trace | `{ trace:rootServiceName = "gateway" }`|
| `trace:id` | string | trace id using hex string | `{ trace:id = "1234567890abcde" }` |
| `event:name` | string | name of event | `{ event:name = "exception" }` |
| `event:timeSinceStart` | duration | time of event in relation to the span start time | `{ event:timeSinceStart > 2ms}` |
| `link:spanID` | string | link span id using hex string | `{ link:spanID = "0000000000000001" }` |
| `link:traceID` | string | link trace id using hex string | `{ link:traceID = "1234567890abcde" }` |

<!-- instrumentation scope isn't included in the 2.6 documentation
| `instrumentation:name` | string | instrumentation scope name | `{ instrumentation:name = "grpc" }` |
| `instrumentation:version`| string | instrumentation scope version | `{ instrumentation:version = "1.0.0" }`|
-->

The trace-level intrinsics, `trace:duration`, `trace:rootName`, and `trace:rootService`, are the same for all spans in the same trace.
Additionally, these intrinsics are significantly more performant because they have to inspect much less data then a span-level intrinsic.
They should be preferred whenever possible to span-level intrinsics.

Expand All @@ -109,15 +138,24 @@ This example searches all Kubernetes clusters called `service-name` that have a
{ resource.k8s.cluster.name="service-name" && trace:rootName !~ ".*perf.*"}
```


### Attribute fields

TraceQL has four different attribute scopes: span attributes, resource attributes, event attributes, and link attributes. By expanding a span in the Grafana UI, you can see both its span attributes (1 in the screenshot) and resource attributes (2 in the screenshot).
TraceQL has four different attribute scopes: span attributes, resource attributes, event attributes, and link attributes.
<!-- instrumentation scope isn't in 2.6 >
instrumentation scope attributes. -->

By expanding a span in the Grafana UI, you can see both its span attributes (1 in the screenshot) and resource attributes (2 in the screenshot).

<p align="center"><img src="assets/span-resource-attributes.png" alt="Example of span and resource attributes." /></p>

Attribute fields are derived from the span and can be customized. Process and span attribute types are [defined by the attribute itself](https://github.com/open-telemetry/opentelemetry-proto/blob/b43e9b18b76abf3ee040164b55b9c355217151f3/opentelemetry/proto/common/v1/common.proto#L30-L38), whereas intrinsic fields have a built-in type. You can refer to dynamic attributes (also known as tags) on the span or the span's resource.
Attribute fields are derived from the span and can be customized.
Process and span attribute types are [defined by the attribute itself](https://github.com/open-telemetry/opentelemetry-proto/blob/b43e9b18b76abf3ee040164b55b9c355217151f3/opentelemetry/proto/common/v1/common.proto#L30-L38), whereas intrinsic fields have a built-in type.
You can refer to dynamic attributes (also known as tags) on the span or the span's resource.

Attributes in a query start with a span scope (for example, `span.http`) or resource scope (for example, `resource.namespace`) depending on what you want to query. This provides significant performance benefits because it allows Tempo to only scan the data you are interested in.
Attributes in a query start with a span, resource, event, or link scope.
For example, you could use `span.http` or `resource.namespace`, depending on what you want to query.
This provides significant performance benefits because it allows Tempo to only scan the data you are interested in.

To find traces with the `GET HTTP` method, your query could look like this:

Expand All @@ -126,6 +164,7 @@ To find traces with the `GET HTTP` method, your query could look like this:
```

For more information about attributes and resources, refer to the [OpenTelemetry Resource SDK](https://opentelemetry.io/docs/reference/specification/resource/sdk/).

#### Examples

Find traces that passed through the `production` environment:
Expand All @@ -138,15 +177,29 @@ Find any database connection string that goes to a Postgres or MySQL database:
{ span.db.system =~ "postgresql|mysql" }
```

You can use the `event` scope to query events that happen within a span.
A span event is a unique point in time during the span’s duration.
While spans help build the structural hierarchy of your services, span events can provide a deeper level of granularity to help debug your application faster and maintain optimal performance.
To learn more about how you can use span events, read the [What are span events?](https://grafana.com/blog/2024/08/15/all-about-span-events-what-they-are-and-how-to-query-them/) blog post.

You can query for an exception in your span event:
```
{ event.exception.message =~ ".*something went wrong.*" }
```

If you've instrumented your traces for span links, you can use the `link` scope to query the link data. A span link associates one span with one or more other spans that are a casual relationship.
For more information on span links, refer to the [Span Links](https://opentelemetry.io/docs/concepts/signals/traces/#span-links) documentation in the Open Telemetry project.

You can search for an attribute in your link:
```
{ link.opentracing.ref_type = "child_of" }
```
<!-- instrumentation scope isn't included in the 2.6 release
Find instrumentation scope programming language:
```
{ instrumentation.language = "java" }
```
-->

### Unscoped attribute fields

Expand Down

0 comments on commit 5db9717

Please sign in to comment.