diff --git a/docs/sources/tempo/api_docs/metrics-summary.md b/docs/sources/tempo/api_docs/metrics-summary.md index 27d51bba429..9f0933b5a85 100644 --- a/docs/sources/tempo/api_docs/metrics-summary.md +++ b/docs/sources/tempo/api_docs/metrics-summary.md @@ -21,7 +21,7 @@ This API returns RED metrics (span count, erroring span count, and latency infor ## Configuration To enable the experimental metrics summary API, you must turn on the local blocks processor in the metrics generator. -Be aware that the generator will use considerably more resources, including disk space, if it is enabled: +Be aware that the generator uses considerably more resources, including disk space, if it's enabled: ```yaml overrides: diff --git a/docs/sources/tempo/configuration/_index.md b/docs/sources/tempo/configuration/_index.md index eaf1bf2476e..2a90318cf7d 100644 --- a/docs/sources/tempo/configuration/_index.md +++ b/docs/sources/tempo/configuration/_index.md @@ -433,6 +433,11 @@ query_frontend: # (default: 2) [max_retries: ] + # The number of goroutines dedicated to consuming, unmarshalling and recombining responses per request. This + # same parameter is used for all endpoints. + # (default: 10) + [response_consumers: ] + # Maximum number of outstanding requests per tenant per frontend; requests beyond this error with HTTP 429. # (default: 2000) [max_outstanding_per_tenant: ] @@ -709,10 +714,10 @@ For more information on configuration options, refer to [this file](https://gith ### Local storage recommendations While you can use local storage, object storage is recommended for production workloads. -A local backend will not correctly retrieve traces with a distributed deployment unless all components have access to the same disk. +A local backend won't correctly retrieve traces with a distributed deployment unless all components have access to the same disk. Tempo is designed for object storage more than local storage. -At Grafana Labs, we have run Tempo with SSDs when using local storage. +At Grafana Labs, we've run Tempo with SSDs when using local storage. Hard drives haven't been tested. You can estimate how much storage space you need by considering the ingested bytes and retention. @@ -876,6 +881,11 @@ storage: # enable to use path-style requests. [forcepathstyle: ] + # Optional. + # Enable to use dualstack endpoint for DNS resolution. + # Check out the (S3 documentation on dualstack endpoints)[https://docs.aws.amazon.com/AmazonS3/latest/userguide/dual-stack-endpoints.html] + [enable_dual_stack: ] + # Optional. Default is 0 # Example: "bucket_lookup_type: 0" # options: 0: BucketLookupAuto, 1: BucketLookupDNS, 2: BucketLookupPath @@ -1246,7 +1256,7 @@ See below for how to override these limits globally or per tenant. #### Standard overrides -You can create an `overrides` section to configure new ingestion limits that applies to all tenants of the cluster. +You can create an `overrides` section to configure ingestion limits that apply to all tenants of the cluster. A snippet of a `config.yaml` file showing how the overrides section is [here](https://github.com/grafana/tempo/blob/a000a0d461221f439f585e7ed55575e7f51a0acd/integration/bench/config.yaml#L39-L40). ```yaml @@ -1540,7 +1550,7 @@ overrides: These tenant-specific overrides are stored in an object store and can be modified using API requests. User-configurable overrides have priority over runtime overrides. -See [user-configurable overrides]{{< relref "../operations/user-configurable-overrides" >}} for more details. +Refer to [user-configurable overrides]{{< relref "../operations/user-configurable-overrides" >}} for more details. #### Override strategies diff --git a/docs/sources/tempo/operations/traceql-metrics.md b/docs/sources/tempo/operations/traceql-metrics.md index a5c7aadaf5b..7444f8eff06 100644 --- a/docs/sources/tempo/operations/traceql-metrics.md +++ b/docs/sources/tempo/operations/traceql-metrics.md @@ -25,15 +25,26 @@ For more information about available queries, refer to [TraceQL metrics queries] To use the metrics generated from traces, you need to: * Set the `local-blocks` processor to active in your `metrics-generator` configuration -* Configure a Tempo data source configured in Grafana or Grafana Cloud -* Access Grafana Cloud or Grafana 10.4 +* Configure a Tempo data source in Grafana or Grafana Cloud +* Access Grafana Cloud or Grafana version 10.4 or newer -## Configure the `local-blocks` processor +## Activate and configure the `local-blocks` processor -Once the `local-blocks` processor is enabled in your `metrics-generator` -configuration, you can configure it using the following block to make sure -it records all spans for TraceQL metrics. +To activate the `local-blocks` processor for all users, add it to the list of processors in the `overrides` block of your Tempo configuration. +```yaml +# Global overrides configuration. +overrides: + metrics_generator_processors: ['local-blocks'] +``` + +To configure the processor per tenant, use the `metrics_generator.processor` override. + +For more information about overrides, refer to [Standard overrides]({{< relref "../configuration#standard-overrides" >}}). + +### Configure the processor + +Next, configure the `local-blocks` processor to record all spans for TraceQL metrics. Here is an example configuration: ```yaml @@ -47,6 +58,9 @@ Here is an example configuration: path: /var/tempo/generator/traces ``` +If you configured Tempo using the `tempo-distributed` Helm chart, you can also set `traces_storage` using your `values.yaml` file. Refer to the [Helm chart for an example](https://github.com/grafana/helm-charts/blob/559ecf4a9c9eefac4521454e7a8066778e4eeff7/charts/tempo-distributed/values.yaml#L362). + + Refer to the [metrics-generator configuration]({{< relref "../configuration#metrics-generator" >}}) documentation for more information. ## Evaluate query timeouts @@ -76,7 +90,6 @@ This is different to the default TraceQL maximum time range of 168 hours (7 days {{< /admonition >}} - For example, in a cloud environment, smaller jobs with more concurrency may be desired due to the nature of scale on the backend.