Byte arrays are represented as strings in generated avro schema?

[asmaier]

It looks like byte arrays are represented as strings in the generated Avro schema. When I try to generate a schema from my File class below and use that schema to write an Avro message, I get an JsonMappingException: Not in union ["null",{"type":"array","items":"string"}]: java.nio.HeapByteBuffer[pos=0 lim=9 cap=9]:

    private static class File {
        public String filename = "TestFile.txt";
        public byte[] filedata = "Test Text".getBytes();
    }

    @Test
    public void testAvroSchemaGenerationWithJackson() throws JsonProcessingException {

        ObjectMapper mapper = new ObjectMapper(new AvroFactory());
        mapper.registerModule(new JavaTimeModule());
        mapper.configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false);

        AvroSchemaGenerator visitor = new AvroSchemaGenerator();
        mapper.acceptJsonFormatVisitor(File.class, visitor);
        AvroSchema schema = visitor.getGeneratedSchema();

        ObjectMapper mapper2 = new ObjectMapper(new AvroFactory());
        System.out.println(mapper2.writer(schema).writeValueAsBytes(new File()));

    }

I believe the reason is, because the generated schema represents the byte array as a string:

{
  "type" : "record",
  "name" : "File",
  "namespace" : "xx.xxx.nss",
  "doc" : "Schema for xx.xxx.SchemaGenerationTest$File",
  "fields" : [ {
    "name" : "filedata",
    "type" : [ "null", {
      "type" : "array",
      "items" : "string"
    } ]
  }, {
    "name" : "filename",
    "type" : [ "null", "string" ]
  } ]
}

How can I make byte arrays be represented simply by the type "bytes" ? Or is this a bug?

[cowtowncoder]

That does sound like a bug, and possibly related to the fact that in JSON byte[] is serialized as Base64-encoded String. But this is not the case for binary formats that have native binary data type.

Thank you for reporting the problem.

[cowtowncoder]

Hmmh. Ok, so the problem is definitely within ByteArraySerializer, reports it as String... will need to see how to resolve.

[cowtowncoder]

Unfortunately it looks like there is no way to fix this reliably for 2.7: neither array route, nor String-with-format work; former because stupid JSON Schema inspired callback route does not have qualifier for number types beyond integer/floating point (hence, can't distinguish short/byte/int/long arrays!), latter because set of formats is fixed.

With 2.8 we can at least add the format, or, alternatively, allow specifying actual element type and not just limited set of pre-defined enums.