Dynamically generating queue functions from Jedis' source code

.github/workflows/tests.yml

@@ -29,6 +29,9 @@ jobs:
         distribution: 'adopt'
         java-version: '11'
 
+    - name: Clone Submodules
+      run: make jedis
+
     - name: Install clojure cli
       uses: DeLaGuardo/setup-clojure@master
       with:

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "vendor/jedis"]
+	path = vendor/jedis
+	url = https://github.com/moclojer/jedis.git

Makefile

@@ -0,0 +1,5 @@
+jedis:
+	git submodule update --init --recursive --remote
+	cd vendor/jedis && make mvn-package-no-tests
+
+all: jedis

README.md

@@ -23,7 +23,148 @@ com.moclojer/rq {:mvn/version "0.x.x"}
 
 > see the versions distributed on clojars
 
-## example
+## building from source
+
+We build Jedis ourselves to enable building queue functions directly using reflection. This approach ensures full compatibility with our library's features.
+
+### prerequisites
+
+- Make sure you have Java JDK (version X.X or higher) installed
+- Ensure you have Make installed on your system
+
+### build steps
+
+1. Clone the repository: `git clone [repository URL]`
+2. Navigate to the project directory: `cd clj-rq`
+3. Run the build command: `make jedis`
+
+After running `make jedis`, the library will be built and ready to be linked with your project. Linking in this context means that the built Jedis library will be properly referenced and used by clj-rq when you include it in your project.
+
+## how clj-rq works under the hood
+
+The `clj-rq` library leverages the `->wrap-method` macro to dynamically generate queue functions by wrapping methods from the Jedis library. This approach ensures that the library is always up-to-date with the latest changes in Jedis, providing enhanced security and compatibility.
+
+The `->wrap-method` macro is defined in `src/com/moclojer/internal/reflection.clj` and is used in `src/com/moclojer/rq/queue.clj` to generate the queue functions. By using reflection, the library can dynamically adapt to changes in the Jedis API, ensuring that the generated functions are always in sync with the underlying Jedis methods.
+
+This dynamic generation process is a key differentiator of the `clj-rq` library, making it more secure and future-proof compared to other libraries that rely on static function definitions.
+
+## functions
+
+This section outlines the key functions available in the clj-rq library, covering both queue and pub/sub operations. For detailed descriptions and examples of each function, please refer to the specific subsections below.
+
+### queue
+
+The `clj-rq` library provides a set of queue functions that are dynamically generated by wrapping methods from the Jedis library. These functions are defined in `src/com/moclojer/rq/queue.clj` and include:
+
+- `push!`: Adds an element to the queue.
+- `pop!`: Removes and returns an element from the queue.
+- `bpop!`: Blocks until an element is available to pop from the queue.
+- `index`: Retrieves an element at a specific index in the queue.
+- `range`: Retrieves a range of elements from the queue.
+- `set!`: Sets the value of an element at a specific index in the queue.
+- `len`: Returns the length of the queue.
+- `rem!`: Removes elements from the queue.
+- `insert!`: Inserts an element into the queue at a specific position.
+- `trim!`: Trims the queue to a specified range.
+
+#### common options
+
+All these functions share common options, such as specifying the queue name and handling encoding/decoding of messages. The options are passed as arguments to the functions and allow for flexible configuration.
+
+#### examples
+
+- **push!**: This function adds an element to the queue. It supports options for specifying the direction (left or right) and encoding the message before pushing it to the queue.
+
+```clojure
+(rq-queue/push! client "my-queue" "message" {:direction :left})
+```
+
+- **pop!**: This function removes and returns an element from the queue. It supports options for specifying the direction (left or right) and decoding the message after popping it from the queue.
+
+```clojure
+(rq-queue/pop! client "my-queue" {:direction :right})
+```
+
+- **bpop!**: This function blocks until an element is available to pop from the queue. It is useful in scenarios where you need to wait for new messages to arrive.
+
+```clojure
+(rq-queue/bpop! client "my-queue" {:timeout 5})
+```
+
+- **index**: This function retrieves an element at a specific index in the queue. It supports options for decoding the retrieved message.
+
+```clojure
+(rq-queue/index client "my-queue" 0)
+```
+
+- **range**: This function retrieves a range of elements from the queue. It supports options for decoding the retrieved messages.
+
+```clojure
+(rq-queue/range client "my-queue" 0 -1)
+```
+
+- **set!**: This function sets the value of an element at a specific index in the queue. It supports options for encoding the message before setting it.
+
+```clojure
+(rq-queue/set! client "my-queue" 0 "new-message")
+```
+
+- **len**: This function returns the length of the queue. It is useful for monitoring the size of the queue.
+
+  ```clojure
+  (rq-queue/len client "my-queue")
+  ```
+
+- **rem!**: This function removes elements from the queue based on a specified pattern. It supports options for specifying the number of elements to remove.
+
+```clojure
+(rq-queue/rem! client "my-queue" "message" {:count 2})
+```
+
+- **insert!**: This function inserts an element into the queue at a specific position. It supports options for encoding the message before inserting it.
+
+```clojure
+(rq-queue/insert! client "my-queue" "pivot-message" "new-message" {:position :before})
+```
+
+- **trim!**: This function trims the queue to a specified range. It is useful for maintaining the size of the queue within certain limits.
+
+```clojure
+(rq-queue/trim! client "my-queue" 0 10)
+```
+
+### pubsub
+
+The `clj-rq` library provides a set of pub/sub functions that facilitate message publishing and subscription in a Redis-backed system. These functions are defined in `src/com/moclojer/rq/pubsub.clj` and include:
+
+- `publish!`: Publishes a message to a specified channel.
+- `group-handlers-by-channel`: Groups message handlers by their associated channels.
+- `create-listener`: Creates a listener that processes messages from subscribed channels.
+- `unarquive-channel!`: Unarchives a channel, making it active again.
+- `pack-workers-channels`: Packs worker channels into a format suitable for processing.
+- `subscribe!`: Subscribes to one or more channels and processes incoming messages.
+
+#### examples
+
+- **publish!**: This function publishes a message to a specified channel. It is used to send messages to subscribers listening on that channel.
+
+```clojure
+(rq-pubsub/publish! client "my-channel" "Hello, World!")
+```
+
+- **subscribe!**: This function subscribes to one or more channels and processes incoming messages using the provided handlers.
+
+```clojure
+(rq-pubsub/subscribe! client ["my-channel"] handlers)
+```
+
+- **unarquive-channel!**: This function unarchives a channel, making it active again. It is useful for reactivating channels that were previously archived.
+
+```clojure
+(rq-pubsub/unarquive-channel! client "my-channel")
+```
+
+## complete example
 
 ```clojure
 (ns rq.example
@@ -34,9 +175,12 @@ com.moclojer/rq {:mvn/version "0.x.x"}
 (def *redis-pool* (rq/create-client "redis://localhost:6379/0"))
 
 ;; queue
-(queue/push! *redis-pool* "my-queue" {:now (java.time.LocalDateTime/now)
-                                      :foo "bar"})
-(println :size (queue/llen *redis-pool* "my-queue"))
+(queue/push! *redis-pool* "my-queue"
+             ;; has to be an array of the elements to push
+             [{:now (java.time.LocalDateTime/now)
+              :foo "bar"}])
+
+(println :size (queue/len *redis-pool* "my-queue"))
 (prn :popped (queue/pop! *redis-pool* "my-queue"))
 
 ;; pub/sub
@@ -50,9 +194,10 @@ com.moclojer/rq {:mvn/version "0.x.x"}
 
 (pubsub/subscribe! *redis-pool* my-workers)
 (pubsub/publish! *redis-pool* "my-channel" "hello world")
-(pubsub/publish! *redis-pool* "my-other-channel" {:my "moclojer team"
-                                                  :data "app.moclojer.com"
-                                                  :hello "maybe you'll like this website"})
+(pubsub/publish! *redis-pool* "my-other-channel"
+                 {:my "moclojer team"
+                 :data "app.moclojer.com"
+                 :hello "maybe you'll like this website"})
 
 (rq/close-client *redis-pool*)
 ```
@@ -86,6 +231,8 @@ sequenceDiagram
     User->>Client: close-client client
     Client-->>Logger: log closing client
     Client-->>User: confirm client closure
-```
+```  
+
+---
 
-Read more about the project [here](docs/README.md).
+Made with 💜 by [moclojer](https://moclojer.com).

deps.edn

@@ -1,8 +1,11 @@
-{:paths   ["src"]
+{:paths ["src" "resources"]
  :deps
- {redis.clients/jedis {:mvn/version "5.1.2"}
+ {redis.clients/jedis {#_#_:mvn/version "5.1.2"
+                       :local/root "vendor/jedis/target/jedis-5.2.0-SNAPSHOT.jar"}
   org.clojure/tools.logging {:mvn/version "1.3.0"}
-  ch.qos.logback/logback-classic {:mvn/version "1.5.6"}}
+  ch.qos.logback/logback-classic {:mvn/version "1.5.6"}
+  camel-snake-kebab/camel-snake-kebab {:mvn/version "0.4.3"}
+  org.clojure/data.json {:mvn/version "2.5.0"}}
 
  :aliases
  {;; clj -A:dev -m com.moclojer.rq

resources/command-allowlist.edn

@@ -0,0 +1,4 @@
+#{"lpush" "rpush" "lpop" "rpop"  "brpop"
+  "blpop" "lrange" "lindex" "lset" "lrem"
+  "llen" "linsert" "ltrim" "rpoplpush"
+  "brpoplpush"  "lmove"}

src/com/moclojer/internal/reflection.clj

@@ -0,0 +1,103 @@
+(ns com.moclojer.internal.reflection
+  (:require
+   [camel-snake-kebab.core :as csk]
+   [clojure.string :as str]
+   [com.moclojer.rq.adapters :as adapters]))
+
+(defn unpack-parameter
+  [parameter]
+  {:type (.. parameter getType getName)
+   :name (csk/->kebab-case (.getName parameter))})
+
+(defn unpack-method
+  [method]
+  {:name (csk/->kebab-case (.getName method))
+   :parameters (map unpack-parameter (.getParameters method))})
+
+(defn underload-methods
+  "Given a list of overloaded `methods`, returns each one's parameter
+  list that matches given its `paramlist`."
+  [paramlist methods]
+  (reduce
+   (fn [underloaded-methods {:keys [name parameters]}]
+     (let [allowed-params (get paramlist name)
+           param-names (map :name parameters)]
+       (if (and (= (count parameters) (count allowed-params))
+                (every? #(some #{%} param-names) allowed-params))
+         (assoc underloaded-methods name parameters)
+         underloaded-methods)))
+   {} methods))
+
+(defn get-klazz-methods
+  [klazz allowmap]
+  (let [allowlist (set (keys allowmap))
+        paramlist (reduce-kv
+                   (fn [acc name method]
+                     (assoc acc name (second method)))
+                   {} allowmap)]
+    (->> (.getMethods klazz)
+         (map unpack-method)
+         (filter #(contains? allowlist (:name %)))
+         (underload-methods paramlist))))
+
+(defmacro ->wrap-method
+  "Wraps given jedis `method` and its respective `parameters` into a
+  common function for this library, which includes, besides the wrapped
+  function itself, options like key pattern and encoding/decoding."
+  [method parameters allowmap]
+  (let [wrapped-method (clojure.string/replace method #"[`0-9]" "")
+        base-doc (str "Wraps redis.clients.jedis.JedisPooled." wrapped-method)
+        param-syms (map #(-> % :name symbol) parameters)
+        [doc _ enc dec] (get allowmap method ["" nil :none :none])]
+    `(defn ~(symbol method)
+       ~(str base-doc \newline doc)
+
+       ~(-> (into ['client] param-syms)
+            (conj '& 'options))
+
+       (let [~{:keys ['pattern 'encoding 'decoding]
+               :or {'pattern :rq
+                    'encoding enc
+                    'decoding dec}} ~'options
+
+             ~'result ~(->> (reduce
+                             (fn [acc par]
+                               (->> (cond
+                                      (= par 'key)
+                                      `(com.moclojer.rq.adapters/pack-pattern
+                                        ~'pattern ~par)
+
+                                      (some #{'value 'string
+                                              'args 'pivot} [par])
+                                      `(com.moclojer.rq.adapters/encode
+                                        ~'encoding ~par)
+
+                                      :else par)
+                                    (conj acc)))
+                             []
+                             param-syms)
+                            (into [(symbol (str "." wrapped-method)) '@client])
+                            (seq))]
+         (try
+           (com.moclojer.rq.adapters/decode ~'decoding ~'result)
+           (catch ~'Exception ~'e
+             (.printStackTrace ~'e)
+             ~'result))))))
+
+(comment
+  (get-klazz-methods
+   redis.clients.jedis.JedisPooled
+   {"rpop" ["hello" ["key" "count"]  :edn-array :none]})
+
+  (require '[clojure.pprint :refer [pprint]])
+  (let [allowmap {"linsert"   ["Inserts a message into a queue in reference to given pivot"
+                               ["key" "where" "pivot" "value"] :edn :none]}
+        [method parameters] (first
+                             (get-klazz-methods
+                              redis.clients.jedis.JedisPooled
+                              allowmap))]
+    (pprint
+     (macroexpand-1 `(->wrap-method ~method ~parameters ~allowmap))))
+
+  ;;
+  )