cmem-plugin-kafka

This eccenca Corporate Memory plugin allows for sending and receiving messages from Apache Kafka.

Installation

cmemc -c my-cmem admin workspace python install cmem-plugin-kafka

Development

• Run task to see all major development tasks.
• Use pre-commit to avoid errors before commit.
• This repository was created with this copier template.

Running Test

This plugin needs needs running Kafka and Corporate Memory orchestrations:

In order to setup access to your Corporate Memory uses cmemc's config eval command to fill environment variables:

$ eval $(cmemc -c my-cmem config eval)

In order to setup access to your Kafka, write the connection details to the .env file:

$ cat .env
KAFKA_BOOTSTRAP_SERVER=localhost:9093
KAFKA_SECURITY_PROTOCOL=PLAINTEXT

To run a Kafka orchestration locally, you can use task:

$ task custom:kafka:start
task: [custom:kafka:start] docker-compose -f docker/docker-compose.yml up --wait --no-color --force-recreate --renew-anon-volumes

[+] Running 2/2
 ⠿ Container docker-zookeeper-1  Healthy                                          1.1s
 ⠿ Container docker-kafka-1      Healthy                                          1.1s

Having Kafka as well as Corporate Memory in place, run the test suite with task check.

confluent-python installation

ARM based Macs

• make sure brew doctor has no issues
• install with brew install librdkafka
• Provide this in your environment (based on this answer)

export CPATH=/opt/homebrew/include
export LIBRARY_PATH=/opt/homebrew/lib

• test build from source in separate environment with:
  • pip install https://files.pythonhosted.org/packages/fb/16/d04dded73439266a3dbcd585f1128483dcf509e039bacd93642ac5de97d4/confluent-kafka-1.8.2.tar.gz
• then try poetry install

Kafka CLI Utility

kcat (formerly kafkacat) is a command-line utility that you can use to test and debug Apache Kafka® deployments. kcat is an open-source utility, available at https://github.com/edenhill/kcat.

To send messages

docker run -it --rm \
        edenhill/kcat \
                -b kafka-broker:9092 \
                -t test \
                -K: \
                -P 

To consume messages

ocker run -it --rm \
        edenhill/kcat \
           -b kafka-broker:9092 \
           -C \
           -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\n\Partition: %p\tOffset: %o\n--\n' \
           -t test