Kafka Streams API?

[jnicholls]

I was very excited to see the SeaStreamer announcement. I have a preconceived notion of what a Stream Processing API ought to encompass... effectively, what Kafka Streams API offers. Suffice to say, I was disappointed that SeaStreamer is at this point merely an abstraction over streaming data sources such as Kafka/Redis/pipes/etc.

Don't get me wrong, this is an awesome start and good work! But for years I've been thinking about building a pure-Rust version of Kafka Streams API that provides high-level data stream processing capabilities, with a windowing set-based data API like Spark Streaming & Flink, but that is distributed and fully in-process like Kafka Streams and Go's equivalent Goka.

I think if you combine SeaStreamer with streaming-backed processing engine, local persistence/cache for logical processors backed by authoritative event logs from e.g. Kafka, and a DataFusion-like set-based data API on top... that's the key for enabling powerful event-driven architectures. Literally, Kafka Streams, but in Rust instead of JVM.

Ok as I typed this, I found Arroyo. Albeit Arroyo is not primarily offering a programmatic experience for embedding stream processing into Rust applications, so there still may be an opportunity here. And then there is Fluvio, which is moving custom stream processors into the cluster rather than the Kafka Streams approach of being embedded and distributed within streaming applications. But, I'll still post this because I'm curious what SeaStreamer's goal is exactly. Good stuff!

[tyt2y3]

Thank you for initiating a discussion.

I think "Literally, Kafka Streams, but in Rust instead of JVM" is an awesome tagline we would want to say out loudly. However, the software engineering philosophy in Rust is slightly different, that things are more 'micro'. You can already do a lot with futures::stream::Stream. For the elaborate things like stream joining, we might create some reusable components to handle them, but then it will not evolve into an all-encompassing 'framework' like Kafka Streams. It's more likely we will ship many bits and pieces for you to develop a streaming pipeline in Rust, but not an engine.

As for the goals, it's always been "to make programming more flexible and ergonomic". Flexible as in it allows you to use Redis for raw data pruning before storing to Kafka as a source of truth. Ergonomic as in - just look at the amount of boilerplate and async trap we took away.

Another goal is to create an abstract interface libraries can bind to. Think of it as "SQLx, but for Streams" or "JDBC, but for Streams". SeaORM is going to use the SeaStreamer to stream out events, and so users can choose a backend, or make their own.

A side-goal is always "to scratch my own itch". Right now I am interested in program instrumentation and I am going to use SeaStreamer as the building block.