Manually set the Trace ID in a root span

[johnmezzetta]

Hi,

How does one manually set the trace ID in the root span? I am surprised that this common use case is not covered in the docs since it is a fundamental aspect of distributed tracing. I am not looking for fancy convenience methods that parse HTTP headers and transparently load a context with Trace ID..., I am looking for the basic methods one would use to take a Trace ID from a python variable and load it into the root span context.

We have python processes that start spontaneously and that need to propagate a Trace ID. How this trace ID gets into the python process is irrelevant - we simply need a way to take a trace ID and insert it into the root span. An official working code example on how to do this would be appreciated.

P.S. I have noticed questions in the past on this topic, but the code snippets given throw "no such member" type errors, which implies a hacking of non-official API methods?

P.S. 2 The sampling rate needs to also be exposed to the user if we are to go live with this library in a production system. Any examples of that?

[lzchen]

@johnmezzetta

P.S. I have noticed questions in the past on this topic, but the code snippets given throw "no such member" type errors, which implies a hacking of non-official API methods?

Which code snippets are you referring to?

P.S. 2 The sampling rate needs to also be exposed to the user if we are to go live with this library in a production system. Any examples of that?

There is no specific code example yet, but feel free to add one if it is a common case. You can get the rate as a property of the TraceIdRatioBased sampler.

trace.set_tracer_provider(TracerProvider(sampler=TraceIdRatioBased(0.05)))
trace.get_tracer_provider().add_span_processor(
    SimpleSpanProcessor(ConsoleSpanExporter())
)
tracer = trace.get_tracer(__name__)
print(trace.get_tracer_provider().sampler.rate)
for x in range(100):
    with tracer.start_as_current_span("foo"):
        print("Hello world!")

[johnmezzetta]

Thanks for the response Leighton. The code snippets were from a random forum on the internet that no longer appears in google searches. The solution did not work so I looked in the official docs for the solution - and did not see one.

I have studied both the spec and the code (way more work that a regular user should be expected to do, btw) to better understand the issue.

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md

The requirement clearly states that the SpanContext carries the TraceId. This is clearly part of the documented API. And the API
is for end users, not just SDK developers, right? Because the python implementers have not exposed the SpanContext object in the API associated with the SDK. It also states that the SpanContext object is immutable after it is created and the opentelemetry API specificaiton clearly states that Trace ID is settable when constructing the SpanContext object. It seems fairly obvious
that this API is to be exposed to the user of the library (not something that is only available/exposed to library implementers).

Furthermore, the Span object encapsulates the immutable SpanContext object. Most of the examples seem to deal with a Span object so the api should be simple:

1. Create an immutable SpanContext object with the trace ID in the constructor arguments
2. Create a Span Object for the root span with the SpanContext in the constructor arguments

From the specification:

SpanContext
https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#spancontext

The OpenTelemetry SpanContext representation conforms to the W3C TraceContext specification. It contains two identifiers - a TraceId and a SpanId - along with a set of common TraceFlags and system-specific TraceState values.
...
The API MUST implement methods to create a SpanContext. These methods SHOULD be the only way to create a SpanContext. This functionality MUST be fully implemented in the API, and SHOULD NOT be overridable.

Comment: it is interesting to note that as a user I can instantiate a SpanContext object as per the API definition, but it is an Orphan object that is not usable anywhere else in the API/SDK.

Span
https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#span

Spans encapsulate:
...
An immutable SpanContext that uniquely identifies the Span
...

Span Creation
https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#span-creation

There MUST NOT be any API for creating a Span other than with a Tracer.
When a new Span is created from a Context... if there is no Span in the Context, the newly created Span will be a root span

Comment: the set_span_in_context is not associated with the Tracer object, so this is likely non-conformant. Also, the specification does not clearly state how a trace ID should be created in the root span. Its interesting to note that the specification focuses on more advanced implementation issues such as thread local mechanisms to keep the implementation simple looking to the user (and technically robust), but seems to leave out this rather mundane and simple detail. Overall, the spec is fantastic in its approach imho.

Conclusion: The specification does not clearly state how the trace ID is defined in a root span. It is unreasonable that the Trace ID be autogenerated in all cases since a process may need to take the trace ID from HTTP or gRPC headers to insert into the Span/SpanContext. Therefore, the only reasonable interpretation is that the constructor of the Span (Span_) object take an optional SpanContext parameter. This parameter would only be used in the case where the span being constructed is the root span. Here is a rough implementation that I have confirmed as working (modified lines in bold or double asterisk depending on this pages rendering of bold):

https://github.com/open-telemetry/opentelemetry-python/blob/main/opentelemetry-sdk/src/opentelemetry/sdk/trace/__init__.py

@contextmanager
def start_as_current_span(
    self,
    name: str,
    context: Optional[context_api.Context] = None,
**span_context: Optional[trace_api.SpanContext] = None,**
    kind: trace_api.SpanKind = trace_api.SpanKind.INTERNAL,
    attributes: types.Attributes = None,
    links: Sequence[trace_api.Link] = (),
    start_time: Optional[int] = None,
    record_exception: bool = True,
    set_status_on_exception: bool = True,
    end_on_exit: bool = True,
) -> Iterator[trace_api.Span]:
    span = self.start_span(
        name=name,
        context=context,
    **span_context=span_context**

...

def start_span(  # pylint: disable=too-many-locals
    self,
    name: str,
    context: Optional[context_api.Context] = None,
**span_context: Optional[trace_api.SpanContext] = None,**
    kind: trace_api.SpanKind = trace_api.SpanKind.INTERNAL,
    attributes: types.Attributes = None,
    links: Sequence[trace_api.Link] = (),
    start_time: Optional[int] = None,
    record_exception: bool = True,
    set_status_on_exception: bool = True,
) -> trace_api.Span:

    parent_span_context = trace_api.get_current_span(
        context
    ).get_span_context()

    if parent_span_context is not None and not isinstance(
        parent_span_context, trace_api.SpanContext
    ):
        raise TypeError(
            "parent_span_context must be a SpanContext or None."
        )

    # is_valid determines root span
    if parent_span_context is None or not parent_span_context.is_valid:
        **# if we have a candidate for root span use the supplied span
        # context if it exists
        if span_context is not None and span_context.trace_id is not None \
                and span_context.is_valid:
            if span_context.span_id is not None and span_context.is_remote \                        
                    is not None and span_context.is_valid:
                # all positional parameters were specified
                # use the supplied context
                parent_span_context = span_context
                trace_id = span_context.trace_id;
            else:
                # if all positional parameters are not supplied
                # follow the existing pattern and use only the trace ID
                parent_span_context = None
                trace_id = span_context.trace_id**
        else:
            parent_span_context = None
            trace_id = self.id_generator.generate_trace_id()


    else:
        trace_id = parent_span_context.trace_id

https://github.com/open-telemetry/opentelemetry-python/blob/main/opentelemetry-sdk/src/opentelemetry/trace/__init__.py

@contextmanager   
@abstractmethod
def start_span(
    self,
    name: str,
    context: Optional[Context] = None,
    **span_context: Optional[SpanContext] = None,**

...

@contextmanager  # type: ignore
@abstractmethod
def start_as_current_span(
    self,
    name: str,
    context: Optional[Context] = None,
    **span_context: Optional[SpanContext] = None,**

And here is an example of its usage:

   trace.set_tracer_provider(
        TracerProvider(
            resource=Resource.create({SERVICE_NAME: myname})
        )
    )
    jaeger_exporter = JaegerExporter(agent_host_name="localhost", agent_port=6831,)
    trace.get_tracer_provider().add_span_processor(
        BatchSpanProcessor(jaeger_exporter)
    )
    tracer = trace.get_tracer(__name__)

    #create our own SpanContext object containing Trace ID
    from opentelemetry.trace import SpanContext
    myspancontext = SpanContext(0xc62b5a1048b67857310acad3085748e3, None, None)

    #pass the SpanContext in as an argument to span creation - note that it will only apply when creating a root span
    with tracer.start_as_current_span(
        name="Ruleset", span_context=myspancontext) as rootspan:

Can you forward this enhancement suggestion to OpenTelemetry devs or suggest a better way to set the trace ID in a root span?

Thanks,
John

[srikanthccv]

Pretty long thread; didn't read it whole; but Here is an example of how you can manually set span context https://github.com/open-telemetry/opentelemetry-python/blob/main/docs/faq-and-cookbook.rst#manually-setting-span-context. Trace id alone is not enough. If you can elaborate on what you are trying to achieve, it would be easier to guide you through it. Where did you get the trace id from? The serialized span context from HTTP will have other details along with trace id.

[orsinium]

Here is an example of how you can manually set span context

The link is dead, and there are no relevant results for searching the word "manually" in the repo.

[bruce-szalwinski]

faq-and-cookbook moved to https://opentelemetry.io/docs/instrumentation/python/cookbook/ as part of #2453

[adriangb]

@bruce-szalwinski is there a way to do that without setting the current span id?
I'm trying to create a span with the same trace ID as a root span but it's own span id. That is, I want to "artificially" create a sibling span for my root span. This is not a problem for non-root spans (you can just call Tracer.start_span()) but for root spans you end up with different trace IDs.

[johnmezzetta]

Hi Srikanth, Thanks for the answer. Basically, we are trying to stitch together several independent processes (some python, some java) together so that the distributed trace is shared between them. Some of the processes will define the trace ID while others will need to continue the distributed tracing using an already defined trace ID. The method that the already defined trace ID is communicated between processes is varied and non-standard (i.e. not in a gRPC or HTTP header). Also, we will sometimes want to force a trace to not be sampled, but have it appear in the system (i.e. via trace flags). I tried the method you suggested fulfills everything we need and I confirmed that it works. I had actually seen this sample in the cookbook, but discarded it because the spec seemed to indicate that it would cause a “noop” – that it was for situations where a dummy trace was needed. And I could see in the code that it was the only way to set a span context via the SDK. Here is the material in the spec that fooled me: Wrapping a SpanContext in a Span The API MUST provide an operation for wrapping a SpanContext with an object implementing the Span interface. This is done in order to expose a SpanContext as a Span in operations such as in-process Span propagation. If a new type is required for supporting this operation, it SHOULD NOT be exposed publicly if possible (e.g. by only exposing a function that returns something with the Span interface type). If a new type is required to be publicly exposed, it SHOULD be named NonRecordingSpan. The behavior is defined as follows: * GetContext MUST return the wrapped SpanContext. * **IsRecording MUST return false to signal that events, attributes and other elements are not being recorded, i.e. they are being dropped.** **The remaining functionality of Span MUST be defined as no-op operations.** Note: This includes End, so as an exception from the general rule, it is not required (or even helpful) to end such a Span. This functionality MUST be fully implemented in the API, and SHOULD NOT be overridable. Anyways, thanks for pointing out the correct approach. Don’t the above words in bold seem a bit strange given that I can see that the span is recorded in Jaeger? Or maybe that is not what is meant by “recording”? Am I missing something in terms of understanding? Best Regards, John

[codeboten]

One thought that came to mind as I was reading the use-case is that sounds like it would be a good case for creating a custom propagator which could be implemented once in each language the system supports then configured via the existing opentelemetry APIs.

[orsinium]

I'm trying to implement a custom distributed tracing and pass the trace ID over the network. Is it a use case that the library can support? I see how I can extract the trace ID (span.get_span_context().trace_id) and so pass it over the network, but can't find a way to set a custom trace ID for a span (SpanContext is immutable, start_span doesn't accept any relevant arguments).

[Larleyt]

@orsinium, have you found a solution to this?

[orsinium]

Based on this tutorial:
https://opentelemetry.io/docs/instrumentation/python/cookbook/#manually-setting-span-context

I've implemented it like this:
https://github.com/orsinium-labs/walnats/blob/master/walnats/middlewares/_integrations.py#L205-L249

But I haven't tested it yet with a real opentelemetry server, I just guess that it might work.

[Larleyt]

Thanks. It seems like not only trace_id is an important parameter for setting correct SpanContext manually, but all the rest – especially span_id and trace_flags (not sure what they are used for, tho).
So, getting span_id along with trace_id from parent and setting explicit trace_flags=TraceFlags(0x01) did the job for me, child span is now in the parent's trace.