Possible to mark an Activity to NOT be recorded on end?

[jviau]

We are working on instrumenting DurableTask Framework and have a unique use case for a specific span. We need to create and destroy this span multiple times, but only have it recorded on the last time. We know this is counter-intuitive to how spans are supposed to work, but it gives the best experience with DurableTask Orchestrations. A single "end to end" invocation of a DurableTask Orchestration involves going back and forth from the queue multiple times. We call these 'replays'. We do not want to show each replay individually as a span (at least not right now), because while it is more "accurate" it is not very useful. Instead, we want to be creating a span, storing the spanId with the orchestration, then re-hydrating that same span on each replay, with the span only being recorded at the very end of the orchestration.

We thought we could accomplish this by creating the Activity then manually swapping out Activity.Current when we are done and never disposing/stopping the initial activity (until the very end that is). However, we found this relevant text from OTel spec:

Any span that is created MUST also be ended. This is the responsibility of the user. API implementations MAY leak memory or other resources (including, for example, CPU time for periodic work that iterates all spans) if the user forgot to end the span.

This means our approach could lead to memory leaks as we were intending to do just that - create a span but never end it.

The question is, how can we accomplish our goal here? How can we stop the span, but have it not be recorded by the SDK?

[jviau]

Reading the docs, I wonder if it would be sufficient for us to start the span with Activity.IsAllDataRequested = true, but right before dispose we set Activity.IsAllDataRequested = false? Will this work?

[jviau]

Toggling IsAllDataRequested may be problematic for OTel: https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry/Trace/TracerProviderSdk.cs#L188

It appears to basically be the same as not ending the activity at all. Which brings us right back to the memory leak issue.

[jviau]

Realized the comment from OTel spec is talking about API (System.Diagnostics.Activity here) and not SDK (OTel SDK here). So yeah the code I linked above could have a memory leak if someone writes a processor which relies on OnStart and OnEnd always being both called for a given activity, I don't think that should be a blocker from us proceeding with our approach.

[vishweshbankwar]

@jviau - Have you checked https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/trace/extending-the-sdk#filtering-processor?

[jviau]

Ah yes that is a very good point - we can add a processor to filter out our spans when the orchestration is not fully complete. Thank you!