Fix OTEL Exception Message Propagation

[rubvs]

Fix on Issue 13603

The issue is reproduced by:

• Running apm-server locally.
• Sending OTLP data via the Go client sendotlp.
  • Some changes was made to apm-server/systemtest/cmd/sendotlp
  • I'll open a PR with this in apm-server once this PR is approved
• View the document in Table form in Kibana -> Analytics -> Discover

The issue is fixed by:

• Applying the changes from apm-data to apm-server
  • cd apm-server
  • go mod edit -replace=github.com/elastic/apm-data=../apm-data
  • make update
• Redo steps as depicted above.

[carsonip]

https://github.com/elastic/apm-data/pull/341/files#diff-a83af8750cbc259cb07425b435f9f248022e844152a698eb5aca74dcad7b10a0R26

// SetErrorMessage is a modelpb.BatchProcessor that sets the APMEvent.Message
// field for error events.

Have you figured out why it was implemented like that in the first place? If it is a bug, do you mind updating the comment as well?

[rubvs]

https://github.com/elastic/apm-data/pull/341/files#diff-a83af8750cbc259cb07425b435f9f248022e844152a698eb5aca74dcad7b10a0R26

// SetErrorMessage is a modelpb.BatchProcessor that sets the APMEvent.Message
// field for error events.

Have you figured out why it was implemented like that in the first place? If it is a bug, do you mind updating the comment as well?

@carsonip thanks for pushing me to investigate. I assumed it was a know bug. My thinking was that the error message should be in error.message, hence the extra green highlight.

@kruskall if I'm not mistaken, you added this comment, can you maybe elaborate on this please?

[kruskall]

Hey @rubvs 👋

The comment was added in this PR (along with the file): 9b6fe76

Historically those processors lived in https://github.com/elastic/apm-server/. Specifically the SetErrorMessage processor was at https://github.com/elastic/apm-server/blob/cdae75cd13183b4c049a0fd1901e4d6a7b4afd67/model/modelprocessor/errormessage.go

It was added by this PR: elastic/apm-server#5974

The original issue mentioned in the PR is at elastic/apm-server#5971.

Based on the issue description I guess setting message is intended.

---

Looking at the codebase, the event message should be correctly set at

apm-data/input/otlp/logs.go

Line 140 in 81d7764

event.Message = body.AsString()

Could you try to comment out the setErrorMessage method call to check if that is working as intended ? Maybe the solution here is to only run the processor if message is empty since it seems the intent of the original issue was to populate the event.message field.

cc @axw since you opened the original issue/PR.

[axw]

Thanks for the ping @kruskall

I think SetErrorMessage predates our support for OTLP logs, and was implemented only with exception span events in mind. In that case message will not be set at all, prior to SetErrorMessage copying the value across.

The point of setting the message field was to ensure the exceptions show up in the Logs app. Not sure if this is necessary any more, but it was back then.

Perhaps the simplest thing to do would be to change SetErrorMessage to only set message if it's empty.

[axw]

Thanks! I think we should probably remove the expectedLogMessage and expectedExceptionMessage bits, but otherwise LGTM.