Revisit 1024 char limit on label values

[carsonip]

There is a 1024 character (as in unicode characters) limit on string label values, on otlp (see code) and intakev2, because we used to depend on Beats & Fleet to set up mappings which would both set ignore_above: 1024 on keyword fields. As apm-server has migrated to ES apm-data plugin, revisit this limitation on label value.

#14716 is a related issue to handle an inconsistency in enforcing this limit.

[inge4pres]

AFAIR OpenTelemetry is also enforcing some limits on the Attributes key/value length?
Is it true? If yes, how do we discern the APM format, if we'd need to?

[carsonip]

AFAIR OpenTelemetry is also enforcing some limits on the Attributes key/value length?

There is truncation for OTel attributes but default truncation length is unlimited by default: https://opentelemetry.io/docs/specs/otel/common/#attribute-limits

[kruskall]

A few things discovered while working on this:

• labels are not the only fields limited to 1024 as pointed out above
• this is not just otlp. Intake also has fields limited to 1024
• while it's true that apm-data is no longer using ignore_above, the new otel-data plugin is using ignore_above: 1024 in quite a few places

Given the above, I wonder if we should consistently remove the 1024 limit everywhere. WDYT ?

[mlunadia]

I don't have any data on the impact of this that can help me form a useful opinion.

• Is this just changing a default which would be reversible?
• Do we have any customers who have raised an issue about data being truncated with the current limits?

[kruskall]

Is this just changing a default which would be reversible?

it could be reversible but the limit is hardcoded and it would require a new version. Another option could be a user manually configuring a pipeline to limit the field length.

[raultorrecilla]

moving this to it 106