module6-stream-processing

Stream processing with Kafka & ksqlDB

This submodule focuses on various approaches for stream processing, such as:

• Kotlin: Producing and Consuming messages from/to Kafka with Kotlin/Java
• kSQLDB: Stream Processing with kSQLDB (Kafka Streams)
• RisingWave: Stream Processing with RisingWave

Hop into their respective folders for details on how to spin them up.

Note: They all use the same docker-compose.yml to bootstrap the kafka cluster. So read the Up and Running instructions down below on how to set it up.

Tech Stack

• Confluent Kafka
• Confluent Schema Registry
• Confluent REST Proxy
• ksqlDB
• Conduktor Platform
• Docker

Up and Running

Developer Setup

1. Spin up Kafka Broker with one of the following options:

1.1. Kafka Cluster (Single-broker) with KRaft:

docker compose up -d

1.2. Kafka Cluster (Multi-broker) with KRaft:

docker compose -f compose.kraft-multi-broker.yaml up -d

1.3. Kafka Cluster (Multi-broker) with Zookeeper:

docker compose -f compose.zookeeper-multi-broker.yaml up -d

1.4. Redpanda:

docker compose -f compose.redpanda.yaml up -d

2. Access Conduktor Web UI for Kafka

open http://localhost:8080

TODO:

• Single-broker Kafka Cluster (with KRaft)
• Multi-broker Kafka Cluster (with KRaft)
• Multi-broker Kafka Cluster (with Zookeeper)
• Confluent Schema Registry
• Confluent Rest Proxy
• ksqlDB Server and CLI
• Kafka Admin UI: Conduktor Console
• Deploy Kafka Cluster (with Zookeeper or KRaft) on K8s using Helm Charts