No effect of KAFKA_BATCH_NUM_MESSAGES variable.

[psreekrishnan]

Hi There,

Setting KAFKA_BATCH_NUM_MESSAGES to different values has no effect on the adapter. Tried by setting10K (default) up to 50K.

I was looking for a way to avoid the below error by increasing the BATCH_NUM_MESSAGES value hoping that I get more room in the adapter to accommodate more messages in the queue.

`{"error":"Local: Queue full","level":"error","msg":"couldn't produce message in kafka topic prometheus-metrics-pft","time":"2022-04-20T10:37:46Z"}

Configure the below variables other than the broker list.
KAFKA_COMPRESSION=snappy
KAFKA_BATCH_NUM_MESSAGES=30000

Also based on the librdkafka document, "queue.buffering.max.ms" is the delay in milliseconds to wait for messages in the producer queue. I am worried about the default value (set to 0) bypassing the KAFKA_BATCH_NUM_MESSAGES values.

queue.buffering.max.ms	P	0 .. 900000	0	Delay in milliseconds to wait for messages in the producer queue to accumulate before constructing message batches (MessageSets) to transmit to brokers. A higher value allows larger and more effective (less overhead, improved compression) batches of messages to accumulate at the expense of increased message delivery latency.Type: integer

`

Is KAFKA_BATCH_NUM_MESSAGES variable not the right variable to bypass the "Queue full" error?

[palmerabollo]

@psreekrishnan sorry for the late reply. I've not touched that code in months, but here is the mapping between vars and librdkafka properties https://github.com/Telefonica/prometheus-kafka-adapter/blob/8d4d51dce302bb36605e0d8df2ce41d21c779ac2/main.go#L33=

KAFKA_BATCH_NUM_MESSAGES is mapped to "batch.num.messages", not "queue.buffering.max.ms", that is not supported yet, as far as I remember. Feel free to contribute that change if you think it's useful in some use cases.

[manosec]

where you are configuring the environmental variables.