[Feature] Support Binary Payloads in All Outboxes

[iancooper]

Is your feature request related to a problem? Please describe.
Some middleware expects a binary payload (for example RMQ) others need it if you use a binary format (Avro, Protobuf) or schema registry in the payload (Kafka). In addition, if we compress or encrypt your payload, it becomes binary.

Converting that payload to a UTF8 string, and then back again, corrupts the payload. If it is to be stored as a string it needs to be base64 encoded instead. When we store a message to the Outbox, we risk corrupting it if we store it as a UTF8 string.

Brighter treats the message payload as a byte[], so we support raw binary payloads, we should provide support with the Outbox for a binary storage of the message body.

Previously we have used strings to store the message body. This has the convenience that if you do not use binary payloads, you can inspect the message contents in the Outbox more easily.

Describe the solution you'd like
The Outbox needs to be able to support binary payloads. Due to the commonality of code other than the DDL to define the Outbox, it is likely that we can just use offer DDL for binary support, and use a parameter to the constructor when required for supportBinaryPayloads that is set to false by default for backwards compatibility. Then we can document that you need to set this for backwards compatibility. (Note that we have switched the Dynamo Outbox to byte[] so we may need to default to true in that case)

Describe alternatives you've considered
An alternative here is to simply use a base64 string whenever content is binary. It is worth reviewing this option at implementation. The MessageBody now accepts setting a character encoding of raw and setting the type is base64. The problem is knowing that this needs conversion back from base64 not UTF8 in the code, which would require storing the character encoding information from the message body in the Outbox, which we do not currently do.

Additional context
This problem has popped out because of increasing usage of our Outbox, and the use of binary content with them such as with Kafka usage of leading bytes for schema registry meaning all its payloads are effectively binary and the provision of transforms such as Compress/Decompress which will turn our content into a binary format. It would be assumed that use of Avro with Kafka would similarly run into the same problem.

[iancooper]

@preardon @holytshirt @jonnyollifflee Just FYI. I'll pick this up shortly.

[preardon]

@iancooper I didn't get time last night to do what I needed for the Outbox Archiver, I'll see if I can carve some time out this week

[iancooper]

From discussion:

"[11:18, 02/02/2023] Ian Cooper: So it sounds like we need an option for SupportBinaryPayloads on out Outbox, which we default to false. Then we have to use specialise the code so that it determines if you wanted to be able to use binary payloads or not before giving you DDL or running SQL.

(Well really it’s SupportNonUTF8Payloads but that is too ugly).
[11:20, 02/02/2023] Ian Cooper: We definitely don’t want a forked outbox because that will become maintenance hell.

We did used to have tooling for the Outbox that Pender wrote. For folks with binary outboxes it might make sense to support tooling to help them work with binary payloads."