Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

As-a-service, consume topic definitions and act accordingly #7

Closed
wants to merge 24 commits into from

Conversation

solsson
Copy link
Contributor

@solsson solsson commented Dec 13, 2017

An experiment with the Kafka's AdminClient API and with declarative topic and schema management. The sequel to a hack that investigated the Kubernetes "operator" + CRD pattern. Consuming CRDs we'd have to reinvent Kafka's consumer commit concept, so I thought that definitions are better kept in a topic.

atamon and others added 24 commits October 25, 2016 16:04
@solsson
Copy link
Contributor Author

solsson commented Dec 13, 2017

My ambition to manage schemas in topic declarations got dented by the behavior or REST Proxy, which requires producers to have either the full schema or the schema id from Schema Registry. ID:s will naturally vary between environments, depending on the order in which we provision services. Having the actual schema in every (micro)service sort of defeats the purpose of centralized management, in particular because producers are allowed to re-define the schema.

I abandoned this PR for a separate module, https://github.com/Yolean/kafka-topic-manager, that tries more optimistic use of Schema Registry + REST Proxy.

Will try to track overall progress, or lack thereof, in Yolean/kubernetes-kafka#101.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants