Investigate options for JSON encoding of byte arrays

[maurolacy]

@maurolacy cloned issue babylonchain/babylon-contract#133 on 2024-06-06:

We're using a lot of byte arrays as inputs (outputs) to (from) contract execution (query) handlers.

Output byte array fields seems to be handled well, but input fields need a Binary wrapper to behave. That means, having to deserialise the type explicitly (i.e. type annotations required, opaque types, etc.). And, using an internal type along with the API struct with Binary-wrapped types, which is cumbersome.

Since we're using a lot of byte array fields, let's see if there are better alternatives to using the Binary wrapper.

A number of alternatives come to mind:

• Use String fields with hex encoding. We're already doing this, but usually using the hex encoded value as an internal value as well. This is far from ideal, and also space / storage inefficient.

• Try to encode JSON on the client like [1, 2, 3, 4] or similar. This seems to work, but it's probably broken because of some JSON serialisation. This perhaps require extra serde-json-wasm support / extensions to work properly. Update: Perhaps deserialising this is utterly inefficient, and that's why Binary was introduced in the first place.

• ...

[maurolacy]

@maurolacy commented on 2024-06-06:

#121 has some initial support of binary fields for byte arrays.

It seems to me that, as long as we're storing those fields independently (i.e. not as parts of a larger struct) this is a good enough approach, as it:

• Does not require extra handling / care on the client side.

• A simple to_vec(), deref() or & can be used to get access to the wrapped byte array.

• No ancillary internal struct is needed, as using Vec<u8> for storage would work out of the box (for isolated / independent fields, that is).

[maurolacy]

@maurolacy commented on 2024-06-24:

After further investigation / tests, it looks like Binary is just an optional wrapper that can be used to improve compactness / simplicity of dealing with byte arrays.

Byte arrays are supported just fine. At least from the CLI, that is.

The JSON syntax, by example for an execute message, is:

{

  "send_data": {

    "binary": "VGhpcyBpcyBleGFtcGxlIHBsYWluLXRleHQgZGF0YQ==",

    "array": [84, 104, 105, 115, 32, 105, 115, 32, 101, 120, 97, 109, 112, 108, 101, 32, 112, 108, 97, 105, 110, 45, 116, 101, 120, 116, 32, 100, 97, 116, 97]

  }

}

That is, the byte array has to be sent with square brackets, no quotes or double quotes, and unsigned decimal numbers for the bytes.

This works fine from the CLI, so, perhaps the problem is then one of syntax. Or, the Go JSON marshaller generating a different format somehow.

In any case, this will require further tests from a Go client context.

A contract written for these tests is here: https://www.github.com/maurolacy/serde-contract

[maurolacy]

@maurolacy commented on 2024-07-02:

Yes, the problem is the Go json package encoding byte arrays as base64(!). This seems to be common practice, which seems to come from Java.

These devs equate common practice with a kind of standard. They are all wrong in the same way, so, it works for them. And it seems they then think they have created a standard.

Rust devs on the contrary, decided not to follow the pack, and sensibly didn't encode nor decode byte arrays as Base64.

[maurolacy]

Closing after / along with #6.