Is there any way to batch get trace info by trace ids with tempo

[zhouyanbl]

Hello,

We are in the process of migrating from Jaeger UI with Elasticsearch storage to Tempo with S3 storage. Some of our older services rely on fetching trace information from Jaeger UI for analysis. These services typically operate at a maximum of 20 queries per second (qps), with queries spanning a relatively short time range.

During this transition, we noticed that Tempo does not offer a batch API to load trace information, forcing us to send requests individually for each trace ID. Unfortunately, this method frequently triggers an error response: “There are too many awesome requests.”

Is there a way to batch retrieve trace information in Tempo? Additionally, we have observed some interesting metrics that might be relevant, which can be found here: https://github.com/grafana/tempo/discussions/3575

4.15 update: Now we increased the number of Tempo querier pods to 32, but this adjustment seems to have had minimal effect. Additionally, it appears that only one block is being primarily queried at any given time; could there be a rate limit in S3 that we are encountering?
Thank you for your assistance.

[electron0zero]

A single query is split into multiple tiny parts called jobs and sent to querier to be processed. queriers will execute these jobs and fetch the parts of the blocks from backend to read data, and process the query. so a single query from end user can result in lot more queries to backend from querier.

we have tiered cached that allows you to cache things like footers, pages, and the whole query results..tiered cache was first release in tempo 2.4.0, so I recommend upgrading to tempo 2.4.1, see docs for more info: https://grafana.com/docs/tempo/v2.4.x/configuration/#cache

once you have tiered cache enabled, look at the hit rate of your page cache, and scale your cache if it's not getting enough hits, we usually see page cache hit rate in 90% range.

[zhouyanbl]

Thank you very much!we will upgrade to v2.4.x to have a try.

[zhouyanbl]

Additionally, may I ask if this cache is for caching blocks, or is it just caching based on queries? If it's the former, I believe it will work very well. If it's the latter, I suspect it may not be suitable for my scenario, as we can almost be certain that each request's trace ID is unique.

[zhouyanbl]

HI bro,we upgraded it to v2.4 and it worked well now.Thank you for your advice!

[joe-elliott]

There are too many awesome requests.

❤️ this typo. You can increase the queue size in the frontend to reduce occurence of this error, but you will just delay other queries from executing.

query_frontend:
    max_outstanding_per_tenant: <some value>

To answer more queries faster you just have to scale out the query path.

Is there a way to batch retrieve trace information in Tempo?

As Suraj pointed out, that does not currently exist. You can, however, write TraceQL queries to access subsets of traces which answer a lot of the questions you may have. Can you point out the kinds of questions you are trying to answer? We can probably give you a TraceQL query that gives you what you want w/o pulling every trace.

[zhouyanbl]

Thank you! In our scenario, the expected request usually looks like this: We have got some trace IDs and want to get the corresponding service name and operation name for them. The trace IDs are created in a short time, so limiting the time range of the query may be helpful.

[joe-elliott]

traceql does not currently support querying by trace id, but it is on the list to add.

[zhouyanbl]

HI bro,we upgraded it to v2.4 and it worked well now.Thank you for your advice!