Skip to content

Lechindianer/kafkaex_lag_exporter

Repository files navigation

KafkaExLagExporter

This project will collect Kafka consumer lag and provide them via Prometheus.

Metrics

Prometheus is a standard way to represent metrics in a modern cross-platform manner. KafkaExLagExporter exposes several metrics as an HTTP endpoint that can be readily scraped by Prometheus.

kafka_consumergroup_group_topic_sum_lag

Labels: cluster_name, group, topic, consumer_id, member_host

The sum of the difference between the last produced offset and the last consumed offset of all partitions in this topic for this group.

kafka_consumergroup_group_lag

Labels: cluster_name, group, partition, topic, member_host, consumer_id

The difference between the last produced offset and the last consumed offset for this partition in this topic partition for this group.

Start

docker run -ti --net="host" -e KAFKA_BROKERS=redpanda:29092 -p 4000:4000 lechindianer/kafkaex_lag_exporter:0.2.0

Now you can check the exposed metrics at http://localhost:4000.

Configuration

KafkaExLagExporter uses 5 seconds as default interval to update the lags. If you want to configure it to use another value set KAFKA_EX_INTERVAL_MS, i.e.

docker run -ti --net="host" -e KAFKA_BROKERS=redpanda:29092 -e KAFKA_EX_INTERVAL_MS=10000 -p 4000:4000 \
  lechindianer/kafkaex_lag_exporter:0.2.0

Developing

To start the project locally:

KAFKA_BROKERS="localhost:9092" iex -S mix 

There is also a Docker compose file included which will start Kafka, serve Kowl (Web UI for Kafka) and start KafkaexLagExporter:

docker compose up --build

Kowl is served at http://localhost:8080.

Tests

MIX_ENV=test mix test --no-start

Code style

Don't forget to check credo for code violations:

mix credo

This project also leverages the use of typespecs in order to provide static code checking:

mix dialyzer

Links

Source is on Gitlab.

The initial project Kafka Lag Exporter was a huge inspiration for me creating my first real Elixir project. Thank you!

About

Check Kafka consumer group lag with Kafka lag exporter

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published