OpenTelemetry Warn Level

[Modularius]

Problem this feature will solve

When the data pipeline is set to emit OpenTelemetry spans, the only spans emitted at at the info level. Consequently, this results in a large number of spans being emitted even during normal running. Storing such a large number of spans is not feasible for longer than a few hours, making this inefficient for long-term bug testing.

Description of solution

A new set of spans emitted at the warn level should be emitted solely during abnormal behaviour. These should aim to avoid frequent repetition of details likely to be irrelevant to bug testing. The general rule is that the warn span should include enough to be able to reproduce the conditions of the error without necessarily being able to diagnose it. More thorough testing can then be done with tracing at the info level.

The warn spans to emit should include (but not be limited to):

• When frames expire in digitiser-aggregator.
• When frames not belonging to any known run are received by nexus-writer.
• Any fatal error (i.e. anything that results in a call to error!).