Using Same Group ID with Multiple Consumer Groups Causes Problems

[ugrkm]

Description

I use ConsumerGroups to consume kafka messages. I have two instances that run on kubernetes. I give the same group ID to both instances. This is to have fault tolerance (one of the pod goes down, the other should be able to keep consuming). What happens is, two instances run side by side. One of them consumes messages alright. But, when I kill one of the pods, the other one can't keep consuming. My guess is maybe broker only accepts one of the pod's ConsumerGroup and just ignores the other one? Could someone please clarify the behavior?

Versions

Sarama	Kafka	Go
v1.43.2	3.5.1	1.22.2

Configuration

config.Consumer.Offsets.AutoCommit.Enable = false
config.Consumer.Offsets.Initial = sarama.OffsetOldest
config.Consumer.Return.Errors = true

[lbydev]

Does your current topic have only one partition? If so, only one consumer will be able to consume messages. Could you check the number of partitions in your topic?

[puellanivis]

Does your current topic have only one partition? If so, only one consumer will be able to consume messages. Could you check the number of partitions in your topic?

I mean, even if this were the case, it should redistribute partitions to the consumers if one of them fails, right?

[lbydev]

Thank you for your reply. I apologize for the misunderstanding. You are correct that Kafka should redistribute partitions among the remaining consumers in the group when one of them fails.

[github-actions]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur.
Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.