Sync eventhub bufferedproducer does not respect max_wait_time with threads<partitions

[epa095]

• azure-eventhub:
• 5.13.0:
• 3.11:

Describe the bug
We use EventHubProducerClient (not async version) with buffered_mode=True. We noticed that for our 32-partition EH that messages in partition 8-31 always arrived in batches of size 1500, much later than max_wait_time (sometimes days after). Events in the earlier partitions arrived as expected.

We happen to be running this on a 4-core azure container app.

We noticed that if we set buffer_concurrency to 32 then max_wait_time seems respected for all partitions.

I think the problem is that the function check_max_wait_time_worker runs as an infinite loop (in the async version the sleep is awaited, which is an important detail) and is submitted to a shared threadpoolexecutor. If we do not set buffer_concurrency then the default threadpoolexecutor is made, with min(32, os.cpu_count() + 4) = 8 threads. That is also the max amount of concurrent tasks the executor can process (even if the task is to sleep), so check_max_wait_time_worker for the higher partition numbers are never executed on the scheduler.

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[kashifkhan]

@epa095 thank you for the feedback. I agree that our docstring and troubleshooting guide should highlight the relationship of buffer concurrency and the number of EventHub partitions. Our general recommendation is that there should be a worker per partition.

[epa095]

@epa095 thank you for the feedback. I agree that our docstring and troubleshooting guide should highlight the relationship of buffer concurrency and the number of EventHub partitions. Our general recommendation is that there should be a worker per partition.

Then I propose that the client by default uses a threadpoolexecutor with one worker per partition. It knows the number of partitions, so it can by default do the right thing.