[RFC] Motivation and design thoughts

[MathiasKoch]

This is our company RFC for the feature, copied here to give people an insight into the thoughts and usecase behind this crate.

Motivation

Currently all logging have been changed to use defmt (https://defmt.ferrous-systems.com/), which makes for a fast debug logging framework.

We should be able to use the two facts about defmt that it is transfer-agnostic and small sized on device to implement a global logger that writes to our external flash instead of RTT.

Background knowledge

NOTE This is slightly outdated, as the format of defmt has been changed to use tags and JSON, but for this project, the description is valid nevertheless.

The way defmtworks is:
Say we have a log statement defmt::info!("This would be awesome to log, but contains a lot of characters that is expensive to include in my binary! This is a variable: {:?}", 5), it would store the string This would be awesome to log, but contains a lot of characters that is expensive to include in my binary! This is a variable: {:?} in a debug section of the ELF, that would not be flashed onto the device, at a known index (in this example index 1).

The log statement on the device would then be changed to send

let bytes = [
    1,  // index
    2,  // timestamp
    5,  // argument
];

through whatever global logger transport is enabled on the device (eg. defmt-rtt)

The timestamp is generated from a function on the device, and could be an actual timestamp (eg. UTC), or just an incrementing number (eg. 1, 2, 3).

A decoder would then be needed to decode these bytes, based on the original ELF containing the aforementioned debug section.

This means that the logging itself is completely transport agnostic (could be serial based, network based, stored and read or whatever. As long as the ELF on the device matches the one used by the decoder)

Proposed Solution

The proposed solution involves writing a defmt-persist crate, that initializes a global logger to write to a given flash area.
The flash area should have an MQTT based API to read the log messages, eg. write to <DeviceType>/log/get/<UUID> and the device will start publishing log statements to <DeviceType>/log/<UUID> or something similar.
Furthermore we need some tooling to read logs and decode them based on the matching elf file from S3.

Implementation Steps

Changes needed to implement the feature:

• Implement defmt-persist. Polished version would be able to rotate logs, so it wouldn't run out of memory. See defmt-rtt and defmt-semihosting for transport layers (https://github.com/knurling-rs/defmt/tree/main/firmware)
• Implement MQTT based device API to read and clear logs. Gold version would be able to filter based on log level and timestamps
• Implement tooling to read and decode messages (https://github.com/knurling-rs/defmt/tree/main/decoder). See (https://github.com/eupn/defmt-print) for a host-side example on how this can be used.

Follow-up task would be to have devices send the full log to MQTT in case of a panic, with a lambda/IoT rule to log it to cloudwatch automatically, so we always persist logs leading up to a panic

[eupn]

@MathiasKoch, now we can tick the third checkbox by including defmt-persist-print tool.

[MathiasKoch]

Sure thing! How do you want to reference it?

[eupn]

@MathiasKoch dunno, let's just include the github repo link in brackets as it's done currently so others can find it when they'd need to decode saved logs.

[MathiasKoch]

Something like that?

[eupn]

LGTM!

[MathiasKoch]

Just opened some issues to anchor some of the potential improvements discussed in your PR.

Did i miss any?

[eupn]

@MathiasKoch

Just opened some issues to anchor some of the potential improvements discussed in your PR.

Did i miss any?

I think no, good job!