Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase session.timeout.ms for KafkaConsumer #358

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

jpbalarini
Copy link

@jpbalarini jpbalarini commented Dec 27, 2018

We found an issue where we were seeing duplicate messages on the bus and some warnings sent by Kafka.

The issue was that the consumers were disconnected and the messages were sent again. The heartbeat_interval_ms on the KafkaConsumer is tightly related to the session_timeout_ms and cannot be set randomly. If no heartbeat is sent before the session timeout occurs, the consumer is considered dead (and currently both are set to the same value, which will cause errors if the timeout occurs before the heartbeat).
As the documentation says https://kafka.apache.org/documentation/ on heartbeat.interval.ms the heartbeat_interval_ms has to be less than 1/3 of the session_timeout_ms. This PR introduces changes to take that into account.

It can be seen on the kafka-python repo that the default values take this into consideration.

Thanks!

@jpbalarini
Copy link
Author

@sibiryakov can we introduce this? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant