Configuration needed for long (wall time) tracing?

[jwineinger]

Is tempo suited for tracing where spans may come in over a relatively long time? I'm thinking specifically about requests that generate async jobs that might take a little while to get scheduled before they run for seconds to minutes. Which means the total time under an hour under normal conditions, and perhaps many hours (due to queue depth) under high load or due to processing outages. Ideally we'd be able to stitch all of that back together to understand the entire flow.

[yvrhdn]

This is not an issue for Tempo. If you have a long running trace, the spans will get spread out over multiple blocks (by default blocks are cut at least every hour but often much faster).
At query time Tempo will search all blocks and combine the result from multiple blocks into a single big trace.

There are however two limitations to keep into account:

• block retention: default retention is 14 days, meaning that blocks older than 14 days get deleted. So spans that ended up in these blocks will also be deleted. This will result in incomplete traces. You can increase block retention (for instance Grafana Cloud goes up to 30 days) but this will require more storage.
• enormous traces: if you are appending a lot of data to a single trace this trace might end up being huge. We have seen this cause issues during querying and compaction. I'm not sure where this limit is, in our internal environments we regularly have traces with over 100k spans and 100 MB of data and they work fine. But I'm guessing multiple 100k spans will become an issue...

[joe-elliott]

I agree with everything here except for the 100MB traces. Between querying, rendering and compacting I think things tend to get hairy when traces cross into the 50/60MB range (uncompressed proto).

[shashankjb]

Hii @joe-elliott In, one of my services I have trace which is producing spans since last 4days. so, it's like a very huge trace and tempo querier fails in such cases. is there any way for tempo to not fail and instead try to show at least the half trace?

[joe-elliott]

This is the default behavior of the v2 traces api endpoint: https://grafana.com/docs/tempo/latest/api_docs/#query-v2

Supported in later versions of Grafana, but I don't know the exact one off the top of my head.

[shashankjb]

Thanks for the clarification