Kafka module can fail to collect partition offsets for topics with many partitions

[jsoriano]

In some Kafka clusters with topics with many partitions there can be problems collecting the offsets, what causes many errors that can flood logs.

This is possibly caused by the way Metricbeat collects the partitions offsets. It gets first the metadata and then it gets the offsets partition by partition. Under some circumstances, with many partitions, some errors happen, like:

• kafka server: Tried to send a message to a replica that is not the leader for some partition. Your metadata is out of date.
• broker is not leader

There are two possible solutions for these errors, but it may be good to implement both:

• Offset requests can request several partition offsets at once. Try to request all the offsets in a single request, or at least split them in less requests. It may be needed to make one request per broker, as the offsets for each partition should be requested to their leaders.
• Handle the error about out of date metadata, and request the metadata again to continue requesting partitions.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[alinazemian]

Is there any workaround for this issue?

[jsoriano]

Is there any workaround for this issue?

@MRaliagha I am afraid that not at the moment 🙁 Are you being hit by this issue? Do you see errors about outdated metadata?

[alinazemian]

@MRaliagha I am afraid that not at the moment 🙁 Are you being hit by this issue? Do you see errors about outdated metadata?

Yes, we are facing this issue with 7.10.1.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[alinazemian]

I'm still having this issue with version 7.16.1.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!