Try to use Apache Kafka clients in idiomatic Clojure.
Kafka has Java/Scala clients API, and programmers have no problem calling them directly in Clojure, so we do not need to "wrap" Java API.
However, from a Clojure programmer’s perspective, direct call Kafka Java API does not feel very good:
-
It is complex, with many interfaces, classes, and methods, which is not a good experience for a dynamic language.
-
It looks more like a framework or even frameworks. Users have to have a lot of background knowledge and implementation details.
-
Clojure programmers could use transducers to deal with data and do not need to use Kafka streams.
Hence, Waterfall is an attempt at a minimalist library with additional optional batteries.
-
Only two core functions:
producerandconsumer. -
Relying on the excellent manifold library,
producer, andconsumeris just the implementation of the Manifold stream. -
Minimum necessary dependencies.
-
Clojure version SERDES, in
robertluo.waterfall.shapenamespace. Underlying, give Kafka API byte array SERDES. -
Optional features require optional dependencies. For example, if you want to use nippy as SERDES, you put it in your classpath.
Because all functions have their schemas incorporated, you can get the best development experience if dependency metosin/malli is in your classpath.
Put (malli.dev/start!) in your user.clj will enable clj-kondo to use the schemas when editing.
Powered by fun-map, you can use one single API for accessing a Kafka cluster without any further knowledge:
You can see an example in this easy example notebook.
| namespace | Description |
|---|---|
robertluo.waterfall |
main API |
robertluo.waterfall.shape |
optional Clojure data shape transformation |
| Filename | Description |
|---|---|
deps.edn |
Clojure tools.deps configuration |
tests.edn |
kaocha test runner configuration |
build.clj |
Clojure tools.build building script |
notebook |
Clerk notebooks to demostrate usage, use |
2022, Robertluo