on_producer_error callbacks don't work as expected in v2.5.0+

[daniil-quix]

Problem

Users can provide on_producer_error callbacks to the Application class to catch and customize the handling of the producer errors.

In v2.5.0, we introduced producer delivery callbacks to improve the delivery guarantees, but the errors reported by the broker don't always trigger the on_producer_error.

Another problem is that these callbacks can be triggered by the messages produced to the changelog or repartition topics, and silencing these errors may harm the data consistency (e.g, producing to the changelog topic should not be silenced).

The serialization errors can still be handled by on_producer_error because they happen before producing the message.

Solution

We need to do some research to find the best way to handle new types of errors safely in the on_producer_error callbacks.