diff --git a/README.md b/README.md index 6ddadbe33c4..ae26c99ff52 100644 --- a/README.md +++ b/README.md @@ -10,13 +10,13 @@ Grafana Tempo is an open source, easy-to-use and high-scale distributed tracing backend. Tempo is cost-efficient, requiring only object storage to operate, and is deeply integrated with Grafana, Prometheus, and Loki. -Tempo is Jaeger, Zipkin, Kafka, OpenCensus and OpenTelemetry compatible. It ingests batches in any of the mentioned formats, buffers them and then writes them to Azure, GCS, S3 or local disk. As such it is robust, cheap and easy to operate! +Tempo is Jaeger, Zipkin, Kafka, OpenCensus and OpenTelemetry compatible. It ingests batches in any of the mentioned formats, buffers them and then writes them to Azure, GCS, S3 or local disk. As such, it is robust, cheap and easy to operate! Tempo implements [TraceQL](https://grafana.com/docs/tempo/latest/traceql/), a traces-first query language inspired by LogQL and PromQL. This query language allows users to very precisely and easily select spans and jump directly to the spans fulfilling the specified conditions: -

Tempo Screenshot

+![Tempo data source query editor](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-ed-example-v11-a.png) -## Getting Started +## Getting started - [Get started documentation](https://grafana.com/docs/tempo/latest/getting-started/) - [Deployment Examples](./example) @@ -24,7 +24,7 @@ Tempo implements [TraceQL](https://grafana.com/docs/tempo/latest/traceql/), a tr - [Helm](./example/helm) - [Jsonnet](./example/tk) -## Further Reading +## Further reading To learn more about Tempo, consult the following documents & talks: @@ -34,7 +34,7 @@ To learn more about Tempo, consult the following documents & talks: [tempo_20_announce]: https://grafana.com/blog/2023/02/01/new-in-grafana-tempo-2.0-apache-parquet-as-the-default-storage-format-support-for-traceql/ [traceql-post]: https://grafana.com/blog/2023/02/07/get-to-know-traceql-a-powerful-new-query-language-for-distributed-tracing/ -## Getting Help +## Getting help If you have any questions or feedback regarding Tempo: @@ -49,7 +49,7 @@ Tempo's receiver layer, wire format and storage format are all based directly on Check out the [Integration Guides](https://grafana.com/docs/tempo/latest/guides/instrumentation/) to see examples of OpenTelemetry instrumentation with Tempo. -## Other Components +## Other components ### tempo-vulture [tempo-vulture](https://github.com/grafana/tempo/tree/main/cmd/tempo-vulture) is Tempo's bird themed consistency checking tool. It writes traces to Tempo and then queries them back in a variety of ways. diff --git a/docs/sources/tempo/configuration/_index.md b/docs/sources/tempo/configuration/_index.md index 1d0ce721f73..1273857f5ee 100644 --- a/docs/sources/tempo/configuration/_index.md +++ b/docs/sources/tempo/configuration/_index.md @@ -3,8 +3,6 @@ title: Configure Tempo menuTitle: Configure description: Learn about available options in Tempo and how to configure them. weight: 400 -aliases: -- /docs/tempo/latest/configuration/ --- # Configure Tempo @@ -33,6 +31,9 @@ The Tempo configuration options include: - [Configuration blocks](#configuration-blocks) - [Block config](#block-config) - [Filter policy config](#filter-policy-config) + - [Filter policy](#filter-policy) + - [Policy match](#policy-match) + - [Examples](#examples) - [KVStore config](#kvstore-config) - [Search config](#search-config) - [WAL config](#wal-config) @@ -305,7 +306,7 @@ metrics_generator: # 0 disables heartbeat altogether [heartbeat_period: | default = 5s] - # The heartbeat timeout, after which, the instance is skipped. + # The heartbeat timeout, after which, the instance is skipped. # 0 disables timeout. [heartbeat_timeout: | default = 1m] @@ -316,7 +317,7 @@ metrics_generator: [instance_interface_names: | default = ["eth0", "en0"] ] # Our advertised IP address in the ring, (usefull if the local ip =/= the external ip) - # Will default to the configured `instance_id` ip address, + # Will default to the configured `instance_id` ip address, # if unset, will fallback to ip reported by `instance_interface_names` # (Effected by `enable_inet6`) [instance_addr: | default = auto(instance_id, instance_interface_names)] @@ -445,14 +446,14 @@ metrics_generator: [max_live_traces: ] # Whether server spans should be filtered in or not. - # If enabled, only parent spans or spans with the SpanKind of `server` will be retained + # If enabled, only parent spans or spans with the SpanKind of `server` will be retained [filter_server_spans: | default = true] # Number of blocks that are allowed to be processed concurently [concurrent_blocks: | default = 10] # A tuning factor that controls whether the trace-level timestamp columns are used in a metrics query. - # If a block overlaps the time window by less than this ratio, then we skip the columns. + # If a block overlaps the time window by less than this ratio, then the columns are skipped. # A value of 1.0 will always load the columns, and 0.0 will never load any. [time_overlap_cutoff: | default = 0.2] @@ -1303,7 +1304,7 @@ parquet_dedicated_columns: # type of the attribute. options: string [type: ] - # scope of the attribute. + # scope of the attribute. # options: resource, span [scope: ] ``` diff --git a/docs/sources/tempo/getting-started/assets/apm-overview.png b/docs/sources/tempo/getting-started/assets/apm-overview.png deleted file mode 100644 index f0c6c473849..00000000000 Binary files a/docs/sources/tempo/getting-started/assets/apm-overview.png and /dev/null differ diff --git a/docs/sources/tempo/getting-started/assets/grafana-query.png b/docs/sources/tempo/getting-started/assets/grafana-query.png deleted file mode 100644 index af9886085b6..00000000000 Binary files a/docs/sources/tempo/getting-started/assets/grafana-query.png and /dev/null differ diff --git a/docs/sources/tempo/getting-started/assets/log-search.png b/docs/sources/tempo/getting-started/assets/log-search.png deleted file mode 100644 index 3709cf32534..00000000000 Binary files a/docs/sources/tempo/getting-started/assets/log-search.png and /dev/null differ diff --git a/docs/sources/tempo/getting-started/assets/tempo-ds.png b/docs/sources/tempo/getting-started/assets/tempo-ds.png deleted file mode 100644 index 991efd7c868..00000000000 Binary files a/docs/sources/tempo/getting-started/assets/tempo-ds.png and /dev/null differ diff --git a/docs/sources/tempo/getting-started/assets/tempo-search.png b/docs/sources/tempo/getting-started/assets/tempo-search.png deleted file mode 100644 index 35830b1455c..00000000000 Binary files a/docs/sources/tempo/getting-started/assets/tempo-search.png and /dev/null differ diff --git a/docs/sources/tempo/getting-started/assets/trace_sample.png b/docs/sources/tempo/getting-started/assets/trace_sample.png deleted file mode 100644 index 5c84bb9c3eb..00000000000 Binary files a/docs/sources/tempo/getting-started/assets/trace_sample.png and /dev/null differ diff --git a/docs/sources/tempo/getting-started/assets/trace_search.png b/docs/sources/tempo/getting-started/assets/trace_search.png deleted file mode 100644 index a1ea98af30b..00000000000 Binary files a/docs/sources/tempo/getting-started/assets/trace_search.png and /dev/null differ diff --git a/docs/sources/tempo/getting-started/assets/trace_service_graph.png b/docs/sources/tempo/getting-started/assets/trace_service_graph.png deleted file mode 100644 index 15cef42303e..00000000000 Binary files a/docs/sources/tempo/getting-started/assets/trace_service_graph.png and /dev/null differ diff --git a/docs/sources/tempo/getting-started/assets/trace_to_logs.png b/docs/sources/tempo/getting-started/assets/trace_to_logs.png deleted file mode 100644 index 95e6f1b645c..00000000000 Binary files a/docs/sources/tempo/getting-started/assets/trace_to_logs.png and /dev/null differ diff --git a/docs/sources/tempo/getting-started/metrics-from-traces.md b/docs/sources/tempo/getting-started/metrics-from-traces.md index 0d786677f97..ee918193175 100644 --- a/docs/sources/tempo/getting-started/metrics-from-traces.md +++ b/docs/sources/tempo/getting-started/metrics-from-traces.md @@ -28,15 +28,15 @@ Span metrics are of particular interest if your system is not monitored with met Metrics generation is disabled by default. Contact Grafana Support to enable metrics generation in your organization. {{% /admonition %}} -After the metrics generator is enabled in your organization, refer to [Metrics-generator configuration]({{< relref "../configuration" >}}) for information about metrics-generator options. +After the metrics-generator is enabled in your organization, refer to [Metrics-generator configuration]({{< relref "../configuration" >}}) for information about metrics-generator options. -

Trace service graph

+![Trace service graph](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-service-graph.png) These metrics exist in your Hosted Metrics instance and can also be easily used to generate powerful custom dashboards.

Trace custom metrics dashboard

-The metrics generator automatically generates exemplars as well which allows easy metrics to trace linking. [Exemplars](/docs/grafana-cloud/monitor-infrastructure/traces/exemplars/) are available in Grafana Cloud so you can also push your own. +The metrics-generator automatically generates exemplars as well which allows easy metrics to trace linking. [Exemplars](https://grafana.com/docs/grafana//fundamentals/exemplars/) are available in Grafana Cloud so you can also push your own.

Trace exemplars

diff --git a/docs/sources/tempo/getting-started/tempo-in-grafana.md b/docs/sources/tempo/getting-started/tempo-in-grafana.md index 48f480326ff..14df9afb2a1 100644 --- a/docs/sources/tempo/getting-started/tempo-in-grafana.md +++ b/docs/sources/tempo/getting-started/tempo-in-grafana.md @@ -16,23 +16,29 @@ Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting The default Tempo search reviews the whole trace. TraceQL provides a method for formulating precise queries so you can quickly identify the traces and spans that you need. Query results are returned faster because the queries limit what is searched. -You can run a TraceQL query either by issuing it to Tempo’s `q` parameter of the [`search` API endpoint]({{< relref "../api_docs#search" >}}), or, for those using Tempo in conjunction with Grafana, by using Grafana’s [TraceQL query editor]({{< relref "../traceql/query-editor" >}}). +You can run a TraceQL query either by issuing it to Tempo’s `q` parameter of the [`search` API endpoint]({{< relref "../api_docs#search" >}}), or, for those using Tempo in conjunction with Grafana, by using the [TraceQL query editor]({{< relref "../traceql/query-editor" >}}). For details about how queries are constructed, read the [TraceQL documentation]({{< relref "../traceql" >}}). -

Query editor showing span results

+![TraceQL query editor showing span results](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-ed-example-v11-a.png) The most basic functionality is to visualize a trace using its ID. Select the TraceQL tab and enter the ID to view it. This functionality is enabled by default and is available in all versions of Grafana. -## Finding traces using Loki logs +## Finding traces using Trace to logs -Traces can be discovered by searching logs for entries containing trace IDs. This is most useful when your application also logs relevant information about the trace that can also be searched, such as HTTP status code, customer ID, etc. This feature requires Grafana 7.5 or later, with a linked Loki data source, and a [traceID derived field](/docs/grafana/latest/datasources/loki/#derived-fields). +Traces can be discovered by searching logs for entries containing trace IDs. +This is most useful when your application also logs relevant information about the trace that can also be searched, such as HTTP status code, customer ID, etc. +This feature requires a linked Loki data source, and a [traceID derived field](/docs/grafana//datasources/loki/#derived-fields). -## Find traces using Tempo tags search +For more information, refer to the [Trace to logs](https://grafana.com/docs/grafana//datasources/tempo/configure-tempo-data-source/#trace-to-logs) documentation. + +![Trace to logs lets you link your tracing data with log data](/media/docs/grafana/data-sources/tempo/trace-to-logs-v11.png) + +## Find traces using Search query builder Search for traces using common dimensions such as time range, duration, span tags, service names, and more. Use the trace view to quickly diagnose errors and high-latency events in your system. -

Showing how to build queries with common dimensions using query builder

+![Showing how to build queries with common dimensions using the query builder](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-builder-v11.png) ### Non-deterministic search @@ -41,7 +47,10 @@ Most search functions are deterministic: using the same search criteria results However, Tempo search is non-deterministic. If you perform the same search twice, you’ll get different lists, assuming the possible number of results for your search is greater than the number of results you have your search set to return. -When performing a search, Tempo does a massively parallel search over the given time range, and takes the first N results. Even identical searches will differ due to things like machine load and network latency. This approach values speed over predictability and is quite simple; enforcing that the search results are consistent would introduce additional complexity (and increase the time the user spends waiting for results). TraceQL follows the same behavior. +When performing a search, Tempo does a massively parallel search over the given time range, and takes the first N results. +Even identical searches differ due to things like machine load and network latency. +This approach values speed over predictability and is quite simple; enforcing that the search results are consistent would introduce additional complexity (and increase the time the user spends waiting for results). +TraceQL follows the same behavior. ## Service graph view @@ -51,7 +60,7 @@ Once the requirements are set up, this pre-configured view is immediately availa For more information, refer to the [service graph view]({{< relref "../metrics-generator/service-graph-view" >}}). -

Service graph view overview

+![Service graph view](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-service-graph.png) ## View JSON file diff --git a/docs/sources/tempo/metrics-generator/_index.md b/docs/sources/tempo/metrics-generator/_index.md index ec5f05f4ab5..632232e097a 100644 --- a/docs/sources/tempo/metrics-generator/_index.md +++ b/docs/sources/tempo/metrics-generator/_index.md @@ -1,7 +1,7 @@ --- aliases: -- /docs/tempo/latest/server_side_metrics/ -- /docs/tempo/latest/metrics-generator/ + - ./server_side_metrics # /docs/tempo//server_side_metrics/ + - /docs/tempo//metrics-generator/ title: Metrics-generator description: Metrics-generator is an optional Tempo component that derives metrics from ingested traces. weight: 500 @@ -17,7 +17,7 @@ The metrics-generator processes spans and writes metrics to a Prometheus data so Enabling metrics generation and remote writing them to Grafana Cloud Metrics produces extra active series that could impact your billing. For more information on billing, refer to [Billing and usage](/docs/grafana-cloud/billing-and-usage/). {{% /admonition %}} -## Overview +## Architecture Metrics-generator leverages the data available in the ingest path in Tempo to provide additional value by generating metrics from traces. @@ -29,9 +29,9 @@ Every processor derives different metrics. Currently, the following processors a - Span metrics - Local blocks -

Service metrics architecture

+

Service metrics architecture

-## Service graphs +### Service graphs Service graphs are the representations of the relationships between services within a distributed system. @@ -39,19 +39,19 @@ This service graphs processor builds a map of services by analyzing traces, with Edges are spans with a parent-child relationship, that represent a jump (e.g. a request) between two services. The amount of request and their duration are recorded as metrics, which are used to represent the graph. -To learn more about this processor, read the [documentation]({{< relref "./service_graphs" >}}). +To learn more about this processor, refer to the [service graph]({{< relref "./service_graphs" >}}) documentation. -## Span metrics +### Span metrics -The span metrics processor derives RED (Request, Error and Duration) metrics from spans. +The span metrics processor derives RED (Request, Error, and Duration) metrics from spans. -The span metrics processor will compute the total count and the duration of spans for every unique combination of dimensions. +The span metrics processor computes the total count and the duration of spans for every unique combination of dimensions. Dimensions can be the service name, the operation, the span kind, the status code and any tag or attribute present in the span. The more dimensions are enabled, the higher the cardinality of the generated metrics. -To learn more about this processor, read the [documentation]({{< relref "./span_metrics" >}}). +To learn more about this processor, refer to the [span metrics]({{< relref "./span_metrics" >}}) documentation. -## Local blocks +### Local blocks The local blocks processor stores spans for a set period of time and enables more complex APIs to perform calculations on the data. The processor must be diff --git a/docs/sources/tempo/metrics-generator/apm-error-example-editor.png b/docs/sources/tempo/metrics-generator/apm-error-example-editor.png index 98e19a23881..a2432d0f39c 100644 Binary files a/docs/sources/tempo/metrics-generator/apm-error-example-editor.png and b/docs/sources/tempo/metrics-generator/apm-error-example-editor.png differ diff --git a/docs/sources/tempo/metrics-generator/apm-error-rate-example.png b/docs/sources/tempo/metrics-generator/apm-error-rate-example.png index 158e6db5137..b267a125b9b 100644 Binary files a/docs/sources/tempo/metrics-generator/apm-error-rate-example.png and b/docs/sources/tempo/metrics-generator/apm-error-rate-example.png differ diff --git a/docs/sources/tempo/metrics-generator/apm-filter-example-numbered.png b/docs/sources/tempo/metrics-generator/apm-filter-example-numbered.png index bd35e2cbdc3..5cf43b72c7d 100644 Binary files a/docs/sources/tempo/metrics-generator/apm-filter-example-numbered.png and b/docs/sources/tempo/metrics-generator/apm-filter-example-numbered.png differ diff --git a/docs/sources/tempo/metrics-generator/apm-filter-example2.png b/docs/sources/tempo/metrics-generator/apm-filter-example2.png index d7a24ddc832..c6bd066b176 100644 Binary files a/docs/sources/tempo/metrics-generator/apm-filter-example2.png and b/docs/sources/tempo/metrics-generator/apm-filter-example2.png differ diff --git a/docs/sources/tempo/metrics-generator/apm-filter-operator.png b/docs/sources/tempo/metrics-generator/apm-filter-operator.png deleted file mode 100644 index 7ad1374859c..00000000000 Binary files a/docs/sources/tempo/metrics-generator/apm-filter-operator.png and /dev/null differ diff --git a/docs/sources/tempo/metrics-generator/apm-overview-numbered.png b/docs/sources/tempo/metrics-generator/apm-overview-numbered.png deleted file mode 100644 index a8ca5394e85..00000000000 Binary files a/docs/sources/tempo/metrics-generator/apm-overview-numbered.png and /dev/null differ diff --git a/docs/sources/tempo/metrics-generator/apm-query-filter-label.png b/docs/sources/tempo/metrics-generator/apm-query-filter-label.png deleted file mode 100644 index 7fbed2de270..00000000000 Binary files a/docs/sources/tempo/metrics-generator/apm-query-filter-label.png and /dev/null differ diff --git a/docs/sources/tempo/metrics-generator/apm-query-filter-value1.png b/docs/sources/tempo/metrics-generator/apm-query-filter-value1.png deleted file mode 100644 index 67f086e7ef5..00000000000 Binary files a/docs/sources/tempo/metrics-generator/apm-query-filter-value1.png and /dev/null differ diff --git a/docs/sources/tempo/metrics-generator/apm-rate-drilldown.png b/docs/sources/tempo/metrics-generator/apm-rate-drilldown.png deleted file mode 100644 index e1cbfb08885..00000000000 Binary files a/docs/sources/tempo/metrics-generator/apm-rate-drilldown.png and /dev/null differ diff --git a/docs/sources/tempo/metrics-generator/apm-remove-filter.png b/docs/sources/tempo/metrics-generator/apm-remove-filter.png deleted file mode 100644 index fcb3656c829..00000000000 Binary files a/docs/sources/tempo/metrics-generator/apm-remove-filter.png and /dev/null differ diff --git a/docs/sources/tempo/metrics-generator/apm-service-graph-drilldown.png b/docs/sources/tempo/metrics-generator/apm-service-graph-drilldown.png deleted file mode 100644 index fac073ff5d7..00000000000 Binary files a/docs/sources/tempo/metrics-generator/apm-service-graph-drilldown.png and /dev/null differ diff --git a/docs/sources/tempo/metrics-generator/apm-service-graph-rows.png b/docs/sources/tempo/metrics-generator/apm-service-graph-rows.png deleted file mode 100644 index 179495c844e..00000000000 Binary files a/docs/sources/tempo/metrics-generator/apm-service-graph-rows.png and /dev/null differ diff --git a/docs/sources/tempo/metrics-generator/apm-service-graph-web.png b/docs/sources/tempo/metrics-generator/apm-service-graph-web.png deleted file mode 100644 index cdff64c0600..00000000000 Binary files a/docs/sources/tempo/metrics-generator/apm-service-graph-web.png and /dev/null differ diff --git a/docs/sources/tempo/metrics-generator/apm-span-metrics.png b/docs/sources/tempo/metrics-generator/apm-span-metrics.png deleted file mode 100644 index 8b8f6e07df0..00000000000 Binary files a/docs/sources/tempo/metrics-generator/apm-span-metrics.png and /dev/null differ diff --git a/docs/sources/tempo/metrics-generator/server-side-metrics-arch-overview.png b/docs/sources/tempo/metrics-generator/server-side-metrics-arch-overview.png deleted file mode 100644 index f38a86283c1..00000000000 Binary files a/docs/sources/tempo/metrics-generator/server-side-metrics-arch-overview.png and /dev/null differ diff --git a/docs/sources/tempo/metrics-generator/service-graph-view.md b/docs/sources/tempo/metrics-generator/service-graph-view.md index 8d5fe249754..25babf08ff2 100644 --- a/docs/sources/tempo/metrics-generator/service-graph-view.md +++ b/docs/sources/tempo/metrics-generator/service-graph-view.md @@ -3,13 +3,13 @@ title: Service graph view menuTitle: Service graph view description: Grafana's service graph view utilizes metrics generated by the metrics-generator (or Grafana Agent) to display span request rates, error rates, and durations, as well as service graphs. aliases: -- /docs/tempo/latest/metrics-generator/app-performance-mgmt + - ./app-performance-mgmt # /docs/tempo//metrics-generator/app-performance-mgmt weight: 400 --- # Service graph view -Grafana's service graph view utilizes metrics generated by the metrics-generator (or Grafana Agent) to display span request rates, error rates, and durations, as well as service graphs. +Grafana's service graph view uses metrics generated by the metrics-generator (or Grafana Alloy) to display span request rates, error rates, and durations, as well as service graphs. Once the requirements are set up, this pre-configured view is immediately available. Using the service graph view, you can: @@ -17,9 +17,9 @@ Using the service graph view, you can: * Discover spans which are consistently erroring and the rates at which they occur * Get an overview of the overall rate of span calls throughout your services * Determine how long the slowest queries in your service take to complete -* Examine all traces that contain spans of particular interest based on rate, error and duration values (RED signals) +* Examine all traces that contain spans of particular interest based on rate, error, and duration values (RED signals) -

Service graph view

+![Service graph view](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-service-graph.png) ## Requirements @@ -41,14 +41,12 @@ Using this view, you can see the top five spans with a type of server (listed in You can refine any of this data using the filters. Selecting any of the data points lets you see more specific data. -The service graph view provides a span metrics visualization (table, screen section 2) and service graph (screen section 3). In addition, you can use the filters (screen section 1) to customize the data displayed. +The service graph view provides a span metrics visualization (table) and service graph (node map). In addition, you can use the filters to customize the data displayed. -

View with numbered sections

- -Any information in the table that has an underline can be selected to show more detailed information. +You can select any information in the table that has an underline to show more detailed information. You can also select any node in the service graph to display additional information. -In the dashboard shown below, the `Ingester.QueryStream` span has a request rate of `144220.22` requests per second. -The `/cortex.Ingester/Query` span has the highest request rate. + +![Service graph with extended informaiton](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-service-graph-prom.png) ### Error rate example @@ -73,8 +71,6 @@ Span metrics generate two metrics: For information about span metrics and how they are calculated, refer to the [Span metrics documentation]({{< relref "../metrics-generator/span_metrics.md" >}}). -

Span metrics table

- ### Table contents The span metrics table contains seven columns with five column headings. Selecting a heading sorts the data by ascending or descending values. @@ -96,11 +92,7 @@ Service graphs infer the topology of a distributed system, provide a high level Service graphs show error rates and latencies, among other relevant data. The service graph layout can be the default or grid. -

Service graph with a connected node layout

- -The grid layout changes the service graph to a series of rows and columns. - -

Service graph with grid layout

+![Node graph view](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-node-graph.png) If you are using the metrics-generator, then it processes traces and generates service graphs in the form of time series metrics like: @@ -119,22 +111,16 @@ However, you can choose which traces are included in the metrics query by filter You can explore data by clicking on selectable items or by using filters. -### Selecting items or nodes for more detail +### Select items or nodes for more detail Clicking on selectable items, such as underlined text in the table or nodes on the service graph, lets you reveal specific details based upon your selection. In the table, you can select items in the **Rate**, **Error Rate**, **Duration (p90)**, and **Links** columns. Choosing one of these items provides details about the span metrics. -

Table with rate drill-down

- - You can view request rate, request histogram, failed request rate, and traces for any node in the service graph. To view more information, select the node in the service graph and then choose an option from the popup. For details on navigating the service graph, refer to the [Node graph panel](/docs/grafana/latest/visualizations/node-graph/) documentation. -

Service graph with drill-down

- - ### Filter with metric queries Using the filters at the top of the screen, you can narrow the data set based upon span attributes (key-value pairs or labels). @@ -143,12 +129,11 @@ You can add one or more label filters. To use the filters: -1. At the top of the Service Graph, select the text box after **Filter** to display a list of available labels. In this case, **server** is selected.
Service graph with grid layout +1. At the top of the Service Graph, select the text box after **Filter** to display a list of available labels. -1. Select or search for a value for the label. In this case, the value of **server** is equal to **tempo-ingester**. The default operator is equals (=).
Select value for label - -1. Optional: Change the operator by selecting **=** and choosing a new option from the drop-down.
Filter operators +1. Select or search for a value for the label. In this case, the value of **server** is equal to **tempo-ingester**. The default operator is equals (=). +1. Optional: Change the operator by selecting **=** and choosing a new option from the drop-down. 1. Optional: Add additional key-value pairs to refine the data set. Any subsequent label filters use AND, which requires both key-value pairs to be presents for matches. @@ -156,9 +141,7 @@ To use the filters: Filters can be removed by selecting the filter drop-down and choosing **– remove filter –**. -

Remove filters

- -In the example below, each field or label represents a key-value pair. Number 1 selects a service as the label whose value is `Go-http-client` (2). The second key-value pair has a client as a label whose value is `02e807`. +Each field or label represents a key-value pair. Number 1 selects a service as the label whose value is `Go-http-client` (2). The second key-value pair has a client as a label whose value is `02e807`.

Filter example with numbers

diff --git a/docs/sources/tempo/metrics-generator/service_graphs/_index.md b/docs/sources/tempo/metrics-generator/service_graphs/_index.md index 6b42c2096f9..490e3a777ac 100644 --- a/docs/sources/tempo/metrics-generator/service_graphs/_index.md +++ b/docs/sources/tempo/metrics-generator/service_graphs/_index.md @@ -1,7 +1,6 @@ --- aliases: -- /docs/tempo/latest/server_side_metrics/service_graphs/ -- /docs/tempo/latest/metrics-generator/service_graphs/ + - ../server_side_metrics/service_graphs/ # /docs/tempo//server_side_metrics/service_graphs/ title: Service graphs description: Service graphs help you understand the structure of a distributed system and the connections and dependencies between its components. weight: 300 @@ -9,6 +8,9 @@ weight: 300 # Service graphs +{{< docs/alias from="/docs/tempo/latest/server_side_metrics/service_graphs/" to="/docs/tempo/latest/metrics-generator/service_graphs/" >}} + + A service graph is a visual representation of the interrelationships between various services. Service graphs help you to understand the structure of a distributed system, and the connections and dependencies between its components: diff --git a/docs/sources/tempo/metrics-generator/tempo-metrics-gen-overview.svg b/docs/sources/tempo/metrics-generator/tempo-metrics-gen-overview.svg new file mode 100644 index 00000000000..4f363e7eb66 --- /dev/null +++ b/docs/sources/tempo/metrics-generator/tempo-metrics-gen-overview.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/sources/tempo/traceql/query-editor/_index.md b/docs/sources/tempo/traceql/query-editor/_index.md index f42f13cfae5..858afaa7d3a 100644 --- a/docs/sources/tempo/traceql/query-editor/_index.md +++ b/docs/sources/tempo/traceql/query-editor/_index.md @@ -3,7 +3,9 @@ title: Write TraceQL queries in Grafana menuTitle: Write TraceQL queries in Grafana description: Learn how to create TraceQL queries in Grafana using the query editor and search. aliases: - - /docs/tempo/latest/traceql/construct-query + - ./construct-query # /docs/tempo//traceql/construct-query + - ./query-editor/traceql-editor # /docs/tempo//traceql/query-editor/traceql-editor + - ./query-editor/traceql-search # /docs/tempo//traceql/query-editor/traceql-search weight: 400 keywords: - Tempo query language @@ -13,26 +15,65 @@ keywords: # Write TraceQL queries in Grafana -You can compose TraceQL queries in Grafana and Grafana Cloud using **Explore** and a Tempo data source. You can use either the **Query type** > **Search** (the TraceQL query builder) or the **TraceQL** tab (the TraceQL query editor). -Both of these methods let you build queries and drill-down into result sets. +The Tempo data source's query editor helps you query and display traces from Tempo in [Explore](https://grafana.com/docs/grafana//explore/). +The queries use [TraceQL](/docs/tempo/latest/traceql), the query language designed specifically for tracing. -To add TraceQL panels to your dashboard, refer to the [Traces panel documentation](/docs/grafana/latest/panels-visualizations/visualizations/traces/). +For general documentation on querying data sources in Grafana, refer to [Query and transform data](/docs/grafana//panels-visualizations/query-transform-data/). -To learn more about Grafana dashboards, refer to the [Use dashboards documentation](/docs/grafana/latest/dashboards/use-dashboards/). +## Before you begin -## TraceQL query builder +You can compose TraceQL queries in Grafana and Grafana Cloud using **Explore** and a Tempo data source. -The TraceQL query builder, located on the **Explore** > **Query type** > **Search** in Grafana, provides drop-downs and text fields to help you write a query. +## Choose a query editing mode -Refer to the [Search using the TraceQL query builder documentation]({{< relref "./traceql-search" >}}) to learn more about creating queries using convenient drop-down menus. +The query editor has three modes, or **Query types**, that you can use to explore your tracing data. +You can use these modes by themselves or in combination to create building blocks to generate custom queries. -![The TraceQL query builder](/static/img/docs/tempo/screenshot-traceql-query-type-search-v10.png) +![The three query types: Search, TraceQL, and Service Graph](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-types.png) +The three **Query types** are: -## TraceQL query editor +- The **Search** query builder provides a user interface for building a TraceQL query. +- The **TraceQL** query editor lets you write your own TraceQL query with assistance from autocomplete. +- The **Service Graph** view displays a visual relationship between services. Refer to the [Service graph view](https://grafana.com/docs/tempo//metrics-generator/service-graph-view/) documentation for more information. -The TraceQL query editor, located on the **Explore** > **TraceQL** tab in Grafana, lets you search by trace ID and write TraceQL queries using autocomplete. +### Search query builder -Refer to the [TraceQL query editor documentation]({{< relref "./traceql-editor" >}}) to learn more about constructing queries using a code-editor-like experience. +The Search query builder provides drop-down lists and text fields to help you write a query. +The query builder is ideal for people who aren't familiar with or want to learn TraceQL. -![The TraceQL query editor](/static/img/docs/tempo/screenshot-traceql-query-editor-v10.png) +Refer to the [Search using the TraceQL query builder documentation](https://grafana.com/docs/grafana//datasources/tempo/query-editor/traceql-search/) to learn more about creating queries using convenient drop-down menus. + +![The Search query builder](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-search-v11.png) + +### TraceQL query editor + +The TraceQL query editor lets you search by trace ID and write TraceQL queries using autocomplete. + +Refer to the [TraceQL query editor documentation](https://grafana.com/docs/grafana//datasources/tempo/query-editor/traceql-editor/) to learn more about constructing queries using a code-editor-like experience. + +![The TraceQL query editor](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-traceql-v11.png) + +You can also search for a Trace ID by entering a trace ID into the query field. + +### Service graph view + +Grafana’s service graph view uses metrics to display span request rates, error rates, and durations, as well as service graphs. +Once the requirements are set up, this pre-configured view is immediately available. + +Using the service graph view, you can: + +- Discover spans which are consistently erroring and the rates at which they occur +- Get an overview of the overall rate of span calls throughout your services +- Determine how long the slowest queries in your service take to complete +- Examine all traces that contain spans of particular interest based on rate, error, and duration values (RED signals) + +For more information about the service graph, refer to [Service graph view](https://grafana.com/docs/tempo//metrics-generator/service-graph-view/). + +![Screenshot of the Service Graph view](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-service-graph.png) + +## Use TraceQL panels in dashboards + +To add TraceQL panels to your dashboard, refer to the [Traces panel documentation](/docs/grafana//panels-visualizations/visualizations/traces/). + +To learn more about Grafana dashboards, refer to the [Use dashboards documentation](/docs/grafana//dashboards/use-dashboards/). \ No newline at end of file diff --git a/docs/sources/tempo/traceql/query-editor/traceql-editor.md b/docs/sources/tempo/traceql/query-editor/traceql-editor.md deleted file mode 100644 index 58b37a46a3b..00000000000 --- a/docs/sources/tempo/traceql/query-editor/traceql-editor.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -description: Learn how to create TraceQL queries in Grafana using the query editor. -keywords: - - grafana - - tempo - - traces - - queries -menuTitle: Write TraceQL queries -title: Write TraceQL queries with the editor ---- - -# Write TraceQL queries with the editor - -[//]: # "Shared content for the TraceQL query editor" -[//]: # "This content is located in /docs/sources/shared/datasources/tempo-editor-traceql.md" - -{{< docs/shared source="grafana" lookup="datasources/tempo-editor-traceql.md" version="" >}} diff --git a/docs/sources/tempo/traceql/query-editor/traceql-search.md b/docs/sources/tempo/traceql/query-editor/traceql-search.md deleted file mode 100644 index 142a493f9eb..00000000000 --- a/docs/sources/tempo/traceql/query-editor/traceql-search.md +++ /dev/null @@ -1,32 +0,0 @@ ---- -description: Learn how to create TraceQL queries in Grafana using Explore > Search. -keywords: - - grafana - - tempo - - traces - - queries -menuTitle: Search traces -title: Search traces using TraceQL query builder ---- - -# Search traces using TraceQL query builder - -Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting traces. -TraceQL provides a method for formulating precise queries so you can zoom in to the data you need. -Query results are returned faster because the queries limit what is searched. - -To learn more about how to query by TraceQL, refer to the [TraceQL documentation](/docs/tempo/latest/traceql). - -The TraceQL query builder, located on the **Explore** > **Query type** > **Search** in Grafana, provides drop-downs and text fields to help you write a query. - -![The TraceQL query builder](/static/img/docs/tempo/screenshot-traceql-query-type-search-v10.png) - -## Enable Search with the query builder - -This feature is automatically available in Grafana 10 (and newer) and Grafana Cloud. - -To enable the TraceQL query builder in self-hosted Grafana through version 10.1, [enable the `traceqlSearch` feature toggle](/docs/grafana/latest/setup-grafana/configure-grafana/feature-toggles/). - -[//]: # "Shared content for the Search - TraceQL query builder" - -{{< docs/shared source="grafana" lookup="datasources/tempo-search-traceql.md" leveloffset="+1" version="" >}}