Skip to content

Commit

Permalink
Typo patch (#19)
Browse files Browse the repository at this point in the history
* mv md to adoc

* adoc format

* fix typo in fn doc
  • Loading branch information
robertluo authored Feb 3, 2023
1 parent 67a3695 commit 5e3b302
Show file tree
Hide file tree
Showing 3 changed files with 74 additions and 69 deletions.
65 changes: 65 additions & 0 deletions README.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
= Waterfall

image:https://github.com/robertluo/waterfall/actions/workflows/main.yml/badge.svg[CI,link=https://github.com/robertluo/waterfall/actions/workflows/main.yml]
image:https://img.shields.io/clojars/v/io.github.robertluo/waterfall.svg[Clojars project,link=https://clojars.org/io.github.robertluo/waterfall]
image:https://cljdoc.org/badge/io.github.robertluo/waterfall[cljdoc,link=https://cljdoc.org/d/io.github.robertluo/waterfall]

== Rational

Try to use Apache Kafka clients in idiomatic Clojure.

Kafka has Java/Scala clients API, and programmers have no problem calling them directly in Clojure, so we do not need to "wrap" Java API.

However, from a Clojure programmer's perspective, direct call Kafka Java API does not feel very good:

- It is complex, with many interfaces, classes, and methods, which is not a good experience for a dynamic language.
- It looks more like a framework or even frameworks. Users have to have a lot of background knowledge and implementation details.
- Clojure programmers could use transducers to deal with data and do not need to use Kafka streams.

Hence, Waterfall is an attempt at a minimalist library with additional optional batteries.

- Only two core functions: `producer` and `consumer`.
- Relying on the excellent https://github.com/clj-commons/manifold[manifold library], `producer`, and `consumer` is just the implementation of the Manifold stream.
- Minimum necessary dependencies.
- Clojure version SERDES, in `robertluo.waterfall.shape` namespace. Underlying, give Kafka API byte array SERDES.
- Optional features require optional dependencies. For example, if you want to use nippy as SERDES, you put it in your classpath.

== Development experience

Because all functions have their schemas incorporated, you can get the best development experience if dependency `metosin/malli` is in your classpath.

Put `(malli.dev/start!)` in your `user.clj` will https://github.com/metosin/malli/blob/master/docs/function-schemas.md#tldr[enable clj-kondo] to use the schemas when editing.

## Single API

Powered by https://github.com/robertluo/fun-map[fun-map], you can use one single API for accessing a Kafka cluster without any further knowledge:

- https://cljdoc.org/d/io.github.robertluo/waterfall/CURRENT/api/robertluo.waterfall#kafka-cluster[`kafka-cluster`]

You can see an example in link:notebook/easy.clj[this easy example notebook].

## API namespaces

[cols="1,1"]
|===
| namespace | Description

| robertluo.waterfall | main API
| robertluo.waterfall.shape | optional Clojure data shape transformation
|===

## Files Description

[cols="1,1"]
|===
| Filename | Description

| deps.edn | Clojure tools.deps configuration
| tests.edn | kaocha test runner configuration
| build.clj | Clojure tools.build building script
| notebook | https://github.com/nextjournal/clerk[Clerk] notebooks to demostrate usage, use `clojure -M:dev:notebook` to use them
|===

== https://unlicense.org[Unlicense]

2022, Robertluo
59 changes: 0 additions & 59 deletions README.md

This file was deleted.

19 changes: 9 additions & 10 deletions src/robertluo/waterfall.clj
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,9 @@
(:require
[robertluo.waterfall
[core :as core]
[util :as util]]
[manifold.stream :as ms]
[robertluo.waterfall.shape :as shape]
[robertluo.waterfall.util :as util]))
[util :as util]
[shape :as shape]]
[manifold.stream :as ms]))

(def schema
"Schema for waterfall"
Expand Down Expand Up @@ -86,13 +85,13 @@
(defn kafka-cluster
"returns a convient life-cycle-map of a kafka cluster. It requires:
- `::nodes` kafka bootstrap servers
- `::shapes` shape of data, example: [(shape/topic (constantly \"sentence\"))(shape/edn)(shape/value-only)]
- `::shapes` shape of data, example: `[(shape/topic (constantly \"sentence\"))(shape/edn)(shape/value-only)]`
If you just use it to publish message,
- optional `::producer-config` can specify additional kafka producer configuration.
If you want to consume from topics:
- `::topics` the topic you want to subscribe to. example: [\"sentence\"]
- `::topics` the topic you want to subscribe to. example: `[\"sentence\"]`
- `::group-id` the group id for the message consumer
- optional `::source-xform` is a transducer to process message before consuming
- optional `::consumer-config` can specify additional kafka consumer configuration. With additions:
Expand All @@ -101,10 +100,10 @@
The returned map has different level of key-values let you use:
- Highest level, no additional knowledge:
- For consumer: `::consume` a function, a one-arity (each message) function as its arg, returns nil.
- For consumer: `::consume` a function, a one-arity (each message) function as its arg, returns `nil`.
- For producer:
-`::put` a function with a message as its arg. e.g. ((::put return-map) {:a 3})
-`::put-all` a function with message sequence as its arg
-`::put!` a function with a message as its arg. e.g. `((::put return-map) {:a 3})`
-`::put-all!` a function with message sequence as its arg
- Mid level, if you need access of underlying manifold sink/source.
- `::source` a manifold source.
- `::sink` a manifold sink.
Expand Down Expand Up @@ -139,7 +138,7 @@
(->> (comp (map (shape/deserializer shapes)) (or source-xform (map identity)))
(xform-source consumer)))
::put! (fnk [::sink] (partial ms/put! sink))
::put-all (fnk [::sink] (partial ms/put-all! sink))
::put-all! (fnk [::sink] (partial ms/put-all! sink))
::consume (fnk [::source] #(ms/consume % source))})
kafka-conf-map)
(throw (ClassNotFoundException. "Need io.github.robertluo/fun-map library in the classpath"))))
Expand Down

0 comments on commit 5e3b302

Please sign in to comment.