Capture span stack trace when the span ends

[SylvainJuge]

Allows to make our otel agent capture span stack traces, this requires:

• define a shared vendor span attribute to store the stack trace and the storage format (text), spec in apm repo OTel attribute for span stacktrace capture apm#828
• define if any filtering is applied on stack trace (for example to remove agent/otel internals from it) no as we plan to store it as-is on backend side, just make the UI reuse it as-is.
• capture the stack trace when the span ends with a span processor, Capture span stack trace #21
• use an ingest-pipeline to draft an apm-server solution, ideally that would allow to reuse the UI without further changes. we decided it wasn't required and could be skipped.
• make apm-server handle this span attribute and store the stack trace in the expected location (or just leave it as an ordinary otel attribute) Map OTel code.stacktrace attribute top-level apm-data#177
• make the UI compatible with the new span attribute, or make sure the feature is not disabled explicitly for Otel agents: [APM] Display plaintext stacktrace on spans kibana#169769
• decide if we need to implement span_stack_trace_min_duration configuration option postponed to Config: Add support for elastic features configuration #22

[SylvainJuge]

Suggestion:

• make apm-data write the stack trace in a "raw" stack trace attribute (not in the labels as it will be by default)
• make the UI rely on this new field when it's available.

[felixbarny]

I'm wondering whether this is actually a feature that we want to port over to the OTel-based agent.

Capturing span stack traces adds overhead in both runtime and storage and I'm not sure if it is even a feature that's used a lot.
I don't have evidence either way (maybe we can get some usage stats from full story) but with the move to OTel, it could be an opportunity to re-evaluate whether we'd even want to offer that feature and whether it should be enabled by default.

[jackshirazi]

We've included it in the list for feature parity with the existing agent, so that our customers can move to the new agent with as few differences as possible. I think we need to include it, but offering it as a feature in the Otel agent is definitely a good suggestion. We can use an alpha implementation in this agent to generate a PR for otel

[SylvainJuge]

@AlexanderWert already opened a PR in semantic conventions open-telemetry/semantic-conventions#435 to add a code.stacktrace attribute, as it has positive feedback so far, it's likely to be the final attribute name. Having this in the semantic conventions allows us to modify the UI in Kibana and rely on labels.code.stacktrace until there is a better option for otel attributes.