Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add partitions and replication-factor #323

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

coderroggie
Copy link
Contributor

Ran into failing tests due to a partition and replication-factor not being in place in the create if not exists clause.

Ran into failing tests due to a partition and replication-factor not being in place in the create if not exists clause.
@solsson
Copy link
Contributor

solsson commented Jun 26, 2020

Kafka 2.5.0, or maybe it was 2.4.0, introduced the option to get those from defaults. Helps a lot when reusing the tests on clusters of different sizes. We should check why that isn't happening.

@coderroggie
Copy link
Contributor Author

Ah.... Well apache/kafka@8e16158 indicates that it should be in 2.4.1 (I'm using the latest in this repo which is 2.4.1). Digging further it appears that https://github.com/apache/kafka/blob/c57222ae8cd7866b7191722656f3065afaee5b7a/core/src/main/scala/kafka/admin/TopicCommand.scala#L674 would indicate that when we are creating and don't have the replicatAssignment and do have the zookeeper option it checks the partitions and replicationFactor options.

I guess beyond that is slightly out of my depth of understanding. https://stackoverflow.com/questions/48986780/kafka-create-topic-with-default-number-of-partitions indicates (without substantiation) that if you have auto.create.topics.enable set then the default partitions should also work... I may be adding that so this all might be a moot point for me, but thought overall it would be good for this test to work for the "defaults" set in this project.

Hope that helps.

@solsson
Copy link
Contributor

solsson commented Jun 27, 2020

I recommend against auto.create.topics.enable. It has caused us too many surprises over the years, because a typo or mistake in a client will create a new topic instead of alerting us through a failure. See 4c389e7.

The CLI args were removed in 7ffbfdc but that commit was vague about version requirements. I started #324 to help us investigate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants