Skip to content

kafka and schema registry basic example in python for PyCon APAC 2022

Notifications You must be signed in to change notification settings

hueiyuan/pycon-kafka

Repository files navigation

pycon-kafka

The repo is kafka and schema registry basic example in python for PyCon APAC 2022

How to use ?

Step 1. Please execute cli to install required libs.

pip3 install -r requirements.txt

Step2. Execute setup.py install to install custom helpers. (> python3.8)

python3 setup.py install

Step3. Please check your kafka and schema registry service is running.

Step4. Please update schema registry auth username and password in .env file.

SCHEMA_REGISTRY_USERNAME=your-schema-registry-username
SCHEMA_REGISTRY_PASSWORD=your-schema-registry-password

Step5. Execute and emulate producer of the kafka.

python3 producer_entrypoint.py --schema-registry-endpoint http://schema-registry-endpoint.xxx.com:8080 --kafka-brokers broker1,broker2

Step6. Execute and emulate consumer of the kafka.

python3 consumer_entrypoint.py --schema-registry-endpoint http://schema-registry-endpoint.xxx.com:8080 --kafka-brokers broker1,broker2 --consumer-group-id your-consumer-group-id

What is the Next?

Since the repo is a basic repo and provide a simple data and one topic of kakfa to emulate. If you are familiar with operate and logics about producer, consumer and schema registry. You can fork this repo and change dynamic data and multiple topics to ingest or consume data to kafka with schema registry.

Reference

  1. Schema Registry Overview
  2. Schema Registry Download and install
  3. confluent kafka client in python
  4. librdkafka configuration

About

kafka and schema registry basic example in python for PyCon APAC 2022

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published