Question: what happens with malformed events?

[MrAnno]

During the discussion about failure metrics in #189, an old memory came to my mind:
Older Windows servers are capable of sending events containing invalid UTF-16 sequences via WEF. This happens because garbage data can be accidentally inserted into the Application container through old WinAPI calls. I have seen production servers occasionally doing this.

My question is: what happens to such a batch in OpenWEC?

(When I had to deal with something similar, I was forced to implement a permissive XML parser and replace/remove the invalid payload; otherwise, the entire batch would have been lost.)

[vruello]

OpenWEC handles two cases that were observed in the wild:

• Invalid UTF16 encoding: by default, OpenWEC replaces malformed sequences with a "replacement character" and issues a warning message (see https://github.com/cea-sec/openwec/blob/main/common/src/encoding.rs#L4, https://docs.rs/encoding_rs/latest/encoding_rs/struct.Decoder.html#method.decode_to_string). This happens quite often in real life situations.
• Truncated events, especially with "RenderedText": this was discussed in Accept events which fail to parse when 'RenderedText' used #46, but to summarize, in some unknown cases, the client may send malformed XML events, leading to an error when parsing XML. Note: This only applies if you are using a formatter that needs to parse the event, so RawJson and Raw are not affected. In this situation, OpenWEC will do its best to recover the raw part of the event and add special "error" fields to the event (see https://github.com/cea-sec/openwec/blob/main/server/src/event.rs#L143, https://github.com/cea-sec/openwec/blob/main/server/src/event.rs#L51).

OpenWEC cannot fail to process a batch of events due to formatting issues. It will always do its best to do it well, but if it can't, some events may be dropped. I believe that it is always better to lose one malformed event rather than to block the client event stream.

[MrAnno]

Perfect. Thank you. 👍🏻