Possibility to produce avro encoded message to schema registry powered topic

[ievgenii-shepeliuk]

Hello

Some tools allow to produce Avro encoded messages to a topic, using schema from schema registry, with pure JSON file as input message.

For example this https://github.com/tchiotludo/akhq

[d-rk]

Hi @ievgenii-shepeliuk,

I'm sorry but I don't see what you are aiming at.
Since we already support avro schemas, can you elaborate a bit on what extension you propose?

Cheers

[eshepelyuk]

Hello @d-rk, sorry for late response, this is a wider explanation of the improvement.

Use case with errors

1. The schema for topic create_user_profile_wallet

{
  "name": "CreateUserProfileWallet",
  "namespace": "Messaging.Contracts.WalletManager.Commands",
  "type": "record",
  "fields": [
    { "name": "CurrencyCode", "type": "string" },
    { "name": "ExpiresOn", "type": ["null", "string"] }
  ]
}

2. First JSON, where ExpiresOn = null

   { 
     "CurrencyCode": "EUR" 
   }
   --

   kafkactl invocation and the output

   kafkactl produce create_user_profile_wallet -f create-null.json -L '--'
   Failed to produce message: failed to convert value to avro data: cannot decode textual record 
   "Messaging.Contracts.WalletManager.Commands.CreateUserProfileWallet": only found 1 of 2 fields    
   

3. Second JSON, where ExpiresOn != null

   {
     "CurrencyCode": "EUR",
     "ExpiresOn": "2022-12-12"
   }
   --

   kafkactl invocation and the output

   kafkactl produce create_user_profile_wallet -f create.json -L '--'
   Failed to produce message: failed to convert value to avro data: cannot decode textual record 
   "Messaging.Contracts.WalletManager.Commands.CreateUserProfileWallet": cannot decode textual union: expected: '{'; actual: '"' for key: "ExpiresOn"
   

So, the messages are not working with kafkactl but those files are valid Avro JSON messages and can be sent to a topic using some other tools, e.g. AKHQ.

Existing solution

To overcome a problem - JSON files must be changed to some unfriendly format.

1. First JSON, where ExpiresOn == null

   {
     "CurrencyCode": "EUR",
     "ExpiresOn": {
       "null": null
     }
   }
   --

2. Second JSON, where ExpiresOn != null

   {
     "CurrencyCode": "EUR",
     "ExpiresOn": {
      "string": "2022-12-12"
    }
   }
   --

Improvement request

As a user I'd like kafkactl to be able to handle plain JSON (as described in the Use case section) for producing Avro encoded message. Without need to convert my plain JSON data to that weird format.

Currently this is working for some other tools, e.g. AKHQ UI, but it is missing for CLI based tools.

[d-rk]

Hey @eshepelyuk

we are using https://github.com/linkedin/goavro for the avro de/serialization.

If you want to omit fields in the json you can specify a default in the schema:

{
  "name": "CreateUserProfileWallet",
  "namespace": "Messaging.Contracts.WalletManager.Commands",
  "type": "record",
  "fields": [
    { "name": "CurrencyCode", "type": "string" },
    { "name": "ExpiresOn", "type": ["null", "string"], "default": null}
  ]
}

With this schema your first use-case should work.

For the second use-case check this issue: linkedin/goavro#106.
In the issue it is mentioned that this is according to the avro spec: https://avro.apache.org/docs/current/spec.html#json_encoding

So I guess there is not much we can do here.

[eshepelyuk]

Hello @d-rk

I've been able to advance on the issue using small python program that leverages fastavro python library.

So my script is able to create AVRO json ( i.e. weird JSON with types ) from plain JSON. Then resulting AVRO JSON can be sent to kafka using kafkactl.

Attaching a new archive containing the script. Plz install fastavro with pip to use it.
json2avro.zip

Script can be used like this

./json2avro.py create-null.json | kafkactl produce create_user_profile_wallet

Could you take a look and check if your libraries support the same functionality, so it can be eventually implemented in kafkactl ?

[d-rk]

Hey @eshepelyuk

I saw that the latest goavro version supports ingesting standard json directly instead of avro json format.

I already tested it out, but sadly it only works for encoding of avro messages and not for the decoding of avro messages.
I filed an issue about that linkedin/goavro#247, but I will probably wait until decoding to standard json also works before integrating it in kafkactl.

[xmcqueen]

I see this mention over here too. Thanks you two.

[eshepelyuk]

Hello
The goavro issue is closed. Any updates on this issue ?

[ievgenii-shepeliuk]

@d-rk Release 3.0 failed, so this feature is not yet released.

[d-rk]