🏫 Awesome Stream Processing 🏫

The term "stream processing" might sound intimidating to many people. We often hear statements like:

• "Stream processing is too difficult to learn and use!" 😱
• "Stream processing is very expensive!" 😱
• "I don’t see any business use cases for stream processing!" 😱

However, we believe this isn't true. ❌

Streaming data is everywhere, generated from operational databases, messaging queues, IoT devices, and many other sources. People can leverage modern stream processing technology to easily address classic real-world problems, using SQL as the programming language.

In this repository, we provide a series of executable demos demonstrating how stream processing can be applied in practical scenarios:

0. Getting started ✅
   • Install Kafka, PostgreSQL, and RisingWave, and run minimal toy examples
1. Querying and processing event streaming data ✅
   • Directly query data stored in event streaming systems (e.g., Kafka, Redpanda)
   • Continuously ingest and analyze data from event streaming systems
2. Bringing analytics closer to operational databases ✅
   • Offload event-driven queries (e.g., materialized views and triggers) from operational databases (e.g., MySQL, PostgreSQL)
3. Real-time ETL (Extract, Transform, Load) ✅
   • Perform ETL continuously and incrementally
4. Solution demonstrations ✅
   • A collection of demos showcasing how stream processing is used in real-world applications
5. Integration demonstrations ✅
   • A collection of Docker Compose demos showcasing the integration between RisingWave and other data platforms. To run these demos, install Docker, pull the RisingWave directory, navigate to the respective folder, and run docker compose up in a terminal window.

We use RisingWave as the default stream processing system to run these demos. We also assume that you have Kafka and/or PostgreSQL installed and possess basic knowledge of how to use these systems. These demos have been verified on Ubuntu and Mac.

All you need is a laptop 💻 - no cluster is required.

Any comments are welcome. Happy streaming!

Join our Slack community to engage in discussions with thousands of stream processing enthusiasts!