Schema Registry and REST Proxy as opt-in folder

[solsson]

Replaces #45 with a 3.0+ & k8s 1.8 merge to master.

Upgrades to latest Confluent Platform, 4.0.0.

Note that the current image was built ignoring junit test exception(s): solsson/dockerfiles@209dccb

[solsson]

I might have misunderstood some fundamentals, but currently these two tools add little value in our stack. My expectation was that Schema Registry would simplify enforcement, and REST Proxy would lower the barrier for microservice-write-to-kafka, trough the JSON-encoded Avro concept.

Schema lookup at runtime is actually not that valuable. In Java services we want specific deserialization with classes generated from our Avro schemas. This means we have to manage schema files anyway (unless we want to use Schema Registry more like our Docker Registry). Libs like https://www.npmjs.com/package/kafka-avro can benefit from runtime lookup, but we already manage the files so why wait until after build?

REST Proxy fails to lower the barrier for schema-encoded writes, because you must provide either the actual schema or its id (correct me if I'm wrong):

• If you provide the schema it will override any existing schema for that topic.
• The ids aren't known at build time (unless, again, we use a central Schema Registry for dev/test/prod).

Might still be useful for cases where a creating a producer or consumer is impractical.
Also, this is a clean opt-in folder and I think it deserves its place in master.

[solsson]

Another argument for merge, even if we use schema files, is https://stackoverflow.com/questions/45635726/kafkaavroserializer-for-serializing-avro-without-schema-registry-url/45637006#45637006

[solsson]

Can we look up a schema id based on a hash (instead of based on the entire schema)? I guess no:

• Schema ID strategy complicates deployment topologies confluentinc/schema-registry#250
• Get canonical avro string from an Avro schema confluentinc/schema-registry#28 (comment)
• Docs: clarify semantics of schema ids? confluentinc/schema-registry#158
• Invalid produce request missing envelope generates 422 with no error message confluentinc/kafka-rest#85

Also, this is a crazy restriction of the REST proxy, while Serdes are fine with String keys (we use UUID a lot) combined with Avro messages:

• Unable to deserialize avro? lensesio/kafka-topics-ui#84 (comment)

[solsson]

I think my disappointment stems from the naming, that Schema Registry and REST Proxy sounds like generic tools when marketed by the Kafka backing company. In fact they are quite biased tools for quite specific use cases, and based on the (in)activity in issues like json-schema support they will stay that way. These biases and use cases are hard, if not impossible, to spot in the documentation.

For example at Yolean we re-create our development environment many times per day. Schema Registry's id handling is hardly compatible with that. Furthermore, we wanted to use String keys with schema-enforced messages, but REST Proxy requires both to be Avro, for reasons unknown.

I've renamed this feature folder avro-tools. REST Proxy has some uses outside the Avro space, but I don't seen any obvious value. KSQL might be a better integration point.

I recommend these tools if your clients are Java, and you want to use Confluent's kafka-streams-avro-serde. But beware of the tradeoffs: Conversion between byte streams and attributes is usually synchrnous and unit testable. Are you prepared to take it online, i.e. introduce runtime dependencies to which there is asynchronous calls? Such calls look to me like the anti-pattern that event-basted architecture ideas argues against. No amount of caching and battle testing (which I'm sure Confluent has done right) can hide this added complexity.

[solsson]

https://www.confluent.io/blog/put-several-event-types-kafka-topic/ is a good read, and hopefully confluentinc/schema-registry#680 will be in next schema registry release.