diff --git a/README.org b/README.org index b5da303..d06866b 100644 --- a/README.org +++ b/README.org @@ -197,7 +197,8 @@ Usage: #+BEGIN_SRC clojure (ns test-db - (:require [konserve.memory :refer [connect-fs-store]] + (:require [#?(:clj konserve.filestore + :cljs konserve.node-filestore) :refer [connect-fs-store]] [konserve.core :as k])) (def my-folder "path/to/folder") @@ -210,18 +211,17 @@ Usage: :END: [[https://developer.mozilla.org/en-US/docs/IndexedDB][IndexedDB]] is provided as reference implementation for -ClojureScript browser backends. +ClojureScript browser backends. The IndexedDB store is restricted to the async api only. Usage: #+BEGIN_SRC clojure (ns test-db - (:require [konserve.memory :refer [connect-idb-store]] - [konserve.core :as k]) - (:require-macros [cljs.core.async.macros :refer [go]])) + (:require [clojure.core.async :refer [go]] + [konserve.indexeddb :refer [connect-idb-store]] + [konserve.core :as k])) - (def dbname "example-db") - (go (def my-db ( + +## Connecting Stores + +```clojure +(require '[konserve.filestore :refer [connect-fs-store]]) + +(def store ( `{:sync? }` + - This is an env map passed around by most functions internally within konserve. The only entry you should typically need to concern yourself with is `:sync?` which is used to control whether functions return channels or values + - an opts map is the last parameter accepted by `konserve.core` functions, but for creating stores, it must be identified by the keyword `:opts` ++ `:config` => map + - this map includes options for manipulating blobs in store specific ways. Very rarely should you ever need to alter the defaults ++ `:buffer-size` => number + - in clj this lets you control the chunk size used for writing blobs. the default is 1mb ++ `:default-serializer` => keyword + - the default serializer is `:FressianSerializer`, but you can override with a keyword identifying a different serializer implementation + - `(connect-store store-name :default-serializer :StringSerializer)` => writes string edn + - jvm also supports `:CBORSerializer` + - you can provide your own serializer by giving a map of `{:MySerializer PStoreSerializerImpl}` to `:serializers` (..see next bullet) and then referencing it via `:default-serializer :MySerializer` + ++ `:serializers` => Map + - this is where you can provide your own serializer to reference via `:default-serializer` + - `konserve.serializers/fressian-serializer` is a convenience function that accepts 2 maps: a map of read-handlers and a map of write-handlers and returns a fressian serializer supporting your custom types + - in clj the handlers are reified `org.fressian.ReadHandlers` & `org.fressian.WriteHandlers` + - see [https://github.com/clojure/data.fressian/wiki/Creating-custom-handlers](https://github.com/clojure/data.fressian/wiki/Creating-custom-handlers) + - in cljs handlers are just functions + - see [https://github.com/pkpkpk/fress](https://github.om/pkpkpk/fress) + ++ `:encryptor` => `{:type :aes :key "s3cr3t"}` + - currently only supports `:aes` in default stores + ++ `:compressor` => `{:type :lz4}` + - currently LZ4 compression is only supported on the jvm + +### Incognito & Records +Konserve intercepts records and writes them as [incognito](https://github.com/replikativ/incognito) tagged literals such that the details of serialization formats are abstracted away and allowing easier interop between different formats. The `:read-handlers` and `:write-handlers` varg args are explicitly meant for working with incognito's tagged literals. + - `:read-handlers` expects an atom wrapping `{'symbol.for.MyRecord map->MyRecord}` for recovering records from incognito tagged literals + - `:write-handlers` expects an atom wrapping `{'symbol.for.MyRecord (fn [record] ..)}` for writing records as incognito tagged literals + - the symbols used in these maps **are not safe for clojurescript** so you should avoid using them + +
+ + +## Working with data +Once you have a store you can access it using `konserve.core` functions. By default functions are asynchronous and return channels yielding values or errors. You can override this by passing an opts map with `:sync? true` + +```clojure +(require '[konserve.core :as k]) + +(k/exists? store :false) ;=> channel + +( true +``` + +You can `get` `get-in` `update` `update-in` and `dissoc` just like a clojure map. + +```clojure +(k/assoc-in store [:fruits :parcha] {:color "yellow" :taste "sour" :quantity 0}) + +(k/update-in store [:fruits :parcha :quantity] inc) + +(k/get-in store [:fruits :parcha :quantity]) ;=> channel<1> + +(k/dissoc store :fruits) +``` + +In the fruits example a simple keyword is the store key, but keys themselves can be arbitrary edn: + +```clojure +(defn memoize-to-store-sync [f] + (fn [& args] + (if-let [result ( channel +``` + +## Working with binary data + +```clojure +(k/bassoc store :blob blob) + +(k/bget store :blob + (fn locked-cb [{is :input-stream}] + (go (input-stream->byte-buffer is)))) ;=> ch +``` +With `bassoc` binary data is written as-is without passing through serialization/encryption/compression. + +`bget` is probably the trickiest function in konserve. It accepts a callback function that is passed a map of `{:input-stream }`. While `locked-cb` is running, konserve locks & holds onto underyling resources (ie file descriptors) until the function exits. You can choose to read from the input stream however you like, but rather than running side-effects within the callback, you should instead return your desired value else the lock will remain held. + ++ when called async, the `locked-cb` should return a channel yielding the desired value that will be read from and yielded by `bget`'s channel ++ in both clojurescript stores, synchronous input streams are not possible. ++ On nodejs you can call `bget` synchronously but the locked-cb will be called with `{:blob js/Buffer}` ++ In the browser with indexedDB, the async only `bget` calls its locked-cb with `{:input-stream :offset }` where offset indicates the amount of bytes to drop before reaching the desired blob start. + - `konserve.indexeddb/read-web-stream` can serve as a locked-cb that will yield a `Uint8Array`. + +## Metadata + +Konserve does some bookkeeping for values by storing them with metadata + +```clojure +(k/get-meta store :key {:sync? true}) +;=> +; {:key :key +; :type +; :last-write } +``` + +## The append log +Konserve provides an append log for writing values quickly. These entries are a special case managed by konserve, where the sequence is stored as a linked list of blobs where each blob is a cons cell of the list. You can name the log with any key, but that key should only be written to or read from using the `append`, `log`, and `reduce-log` functions. + +```clojure +(dotimes [n 6] + ( channel<(0 1 2 3 4 5)> + +(k/reduce-log store :log + (fn [acc n] + (if (even? n) + (conj acc n) + acc)) + []) ;=> channel<[0 2 4]> +``` + +## konserve.gc + +`konserve.gc/sweep!` lets you prune the store based on a whitelist set of keys to keep and a timestamp cutoff before which un whitelisted entries should be deleted + +## konserve.cache + +`konserve.cache/ensure-cache` wraps a store with a lru-cache to avoid hitting external memory for frequently accessed keys diff --git a/karma.conf.js b/karma.conf.js new file mode 100644 index 0000000..6314a8b --- /dev/null +++ b/karma.conf.js @@ -0,0 +1,15 @@ +module.exports = function (config) { + config.set({ + browsers: ['ChromeHeadless'], + basePath: 'target', + files: ['ci.js'], + frameworks: ['cljs-test'], + plugins: ['karma-cljs-test', 'karma-chrome-launcher'], + colors: true, + logLevel: config.LOG_INFO, + client: { + args: ["shadow.test.karma.init"], + singleRun: true + } + }) +}; \ No newline at end of file diff --git a/package.json b/package.json new file mode 100644 index 0000000..3c35ba0 --- /dev/null +++ b/package.json @@ -0,0 +1,13 @@ +{ + "name": "CITests", + "version": "1.0.0", + "description": "Testing", + "devDependencies": { + "karma": "^6.4.2", + "karma-chrome-launcher": "^2.2.0", + "karma-cljs-test": "^0.1.0", + "shadow-cljs": "^2.26.0" + }, + "author": "", + "license": "MIT" +} diff --git a/shadow-cljs.edn b/shadow-cljs.edn index b1ea626..8ae3849 100644 --- a/shadow-cljs.edn +++ b/shadow-cljs.edn @@ -1,7 +1,32 @@ {:deps {:aliases [:cljs]} - :source-paths ["src"] + :dev-http {8021 "out/browser-tests"} :builds {:app {:target :browser :output-dir "public/js" - :modules {:main {:entries [konserve.core]}}}}} + :modules {:main {:entries [konserve.core]}}} + + :node-tests ; shadow-cljs release node-tests && node out/node-tests.js + {:target :node-test + :output-to "out/node-tests.js" + :ns-regexp "^(?!konserve.indexeddb-test)" + :compiler-options {:infer-externs true + :externs ["cljs_node_io/externs.js"] + :closure-warnings {:useless-code :off}}} + + :browser-tests + ; shadow-cljs watch :browser-tests + ; http://localhost:8021/ + {:target :browser-test + :test-dir "out/browser-tests" + :ns-regexp "^(?!konserve.node-filestore-test|konserve.node-filestore)" + :compiler-options {:infer-externs true + :closure-warnings {:useless-code :off}}} + + :ci + {:target :karma + :output-to "target/ci.js" + :ns-regexp "^(?!konserve.node-filestore-test|konserve.node-filestore)" + :compiler-options {:infer-externs true + :compiler-options {:optimizations :advanced} + :closure-warnings {:useless-code :off}}}}} diff --git a/src/konserve/cache.cljc b/src/konserve/cache.cljc index 8fdaaa3..c173191 100644 --- a/src/konserve/cache.cljc +++ b/src/konserve/cache.cljc @@ -9,7 +9,8 @@ #?(:clj [clojure.core.cache :as cache] :cljs [cljs.cache :as cache]) [konserve.core #?@(:clj (:refer [go-locked locked])) :as core] - [konserve.utils :refer [meta-update async+sync *default-sync-translation*]] + [konserve.utils :refer [meta-update #?(:clj async+sync) *default-sync-translation*] + #?@(:cljs [:refer-macros [async+sync]])] [taoensso.timbre :refer [trace]] [superv.async :refer [go-try- lock poll!) + (debug "WARNING: konserve lock is not active. Only use the synchronous variant with the memory store in JavaScript.")))) #_{:clj-kondo/ignore [:clojure-lsp/unused-public-var]} (defmacro locked [store key & code] @@ -246,7 +248,9 @@ (fn [{is :input-stream}] (let [tmp-file (io/file \"/tmp/my-private-copy\")] (io/copy is tmp-file))) - " + + When called asynchronously (by default or w/ {:sync? false}), the locked-cb + must synchronously return a channel." ([store key locked-cb] (bget store key locked-cb {:sync? false})) ([store key locked-cb opts] diff --git a/src/konserve/indexeddb.cljs b/src/konserve/indexeddb.cljs index 256ec32..1f47682 100644 --- a/src/konserve/indexeddb.cljs +++ b/src/konserve/indexeddb.cljs @@ -1,12 +1,11 @@ (ns konserve.indexeddb - (:require-macros [cljs.core.async.macros :refer [go]]) - (:require [cljs.core.async :refer [take! put! close!]] + (:require [clojure.core.async :refer [go take! put! close!]] [konserve.compressor] [konserve.encryptor] [konserve.impl.defaults :as defaults] [konserve.impl.storage-layout :as storage-layout] [konserve.serializers] - [konserve.utils :refer [with-promise]])) + [konserve.utils :refer-macros [with-promise]])) (defn connect-to-idb [db-name] (let [req (js/window.indexedDB.open db-name 1)] @@ -135,11 +134,11 @@ (put! out (ex-info "error reading blob from objectStore" {:cause res :caller 'konserve.indexeddb/read-binary})) - (do - (locked-cb {:input-stream (.stream res) - :size (.-size res) - :offset (+ meta-size storage-layout/header-size)}) - (close! out))))))) + (take! + (locked-cb {:input-stream (.stream res) + :size (.-size res) + :offset (+ meta-size storage-layout/header-size)}) + #(put! out %))))))) (defrecord ^{:doc "buf is cached data that has been read from the db, & {header metadata value} are bin data to be written. @@ -267,9 +266,37 @@ (db-exists? db-name)) (-sync-store [_this env] (when-not (:sync? env) (go)))) +(defn read-web-stream + "Accepts the bget locked callback arg and returns a promise-chan containing + a concatenated byte array with the first offset bytes dropped + `(k/bget store :key read-web-stream)`" + [{:keys [input-stream offset]}] + (let [reader (.getReader input-stream) + chunks #js[]] + (with-promise out + (let [read-chunk (fn read-chunk [] + (.then (.read reader) + (fn [result] + (if (.-done result) + (do + (some->> (.-value result) (.push chunks)) + (if (== 1 (alength chunks)) + (put! out (.slice (aget chunks 0) offset)) + (let [total-length (reduce + (map count chunks)) + final-array (js/Uint8Array. (inc total-length)) + _i (atom 0)] + (doseq [chunk (array-seq chunks)] + (.set final-array chunk @_i) + (swap! _i + (alength chunk))) + (put! out (.slice final-array offset))))) + (do + (.push chunks (.-value result)) + (read-chunk)))) + (fn [err] (put! out err))))] + (read-chunk))))) + (defn connect-idb-store "Connect to a IndexedDB backed KV store with the given db name. - Optional serializer, read-handlers, write-handlers. This implementation stores all values as js/Blobs in an IndexedDB object store instance. The object store itself is nameless, and there @@ -288,15 +315,20 @@ db and core.async gets into a weird state due to an unhandled error, you will be unable to delete the database until the vm is restarted - + As of November 2022 firefox does not support IDBFactory.databases() so + + As of November 2023 firefox does not support IDBFactory.databases() so expect list-dbs, db-exists?, & PBackingStore/-store-exists? to all throw. You must work around this by keeping track of each db name you intend to delete https://developer.mozilla.org/en-US/docs/Web/API/IDBFactory/databases#browser_compatibility - + PBackingBlob/-read-binary returns a webstream that is *not* queued to the - value offset in the same way that the filestore implementations are. Consumers - must discard the amount of bytes found in the :offset key of the locked-cb - arg map. See: https://developer.mozilla.org/en-US/docs/Web/API/Blob/stream" + + `konserve.core/bget` locked-cb arg is given a webstream that is *not* queued + to the value offset in the same way that the filestore implementations are. + See: https://developer.mozilla.org/en-US/docs/Web/API/Blob/stream + - consumers must discard the amount of bytes found in the :offset key of + the locked-cb arg map. These bytes are meta data for konserve and not + part of the value you are retrieving + - `konserve.indexeddb/read-web-stream` will accept the argument to the + locked-cb and return a promise-chan receiving err|bytes at the cost of + allocating a larger array for the chunks to be copied into" [db-name & {:as params}] (let [store-config (merge {:default-serializer :FressianSerializer :compressor konserve.compressor/null-compressor diff --git a/src/konserve/memory.cljc b/src/konserve/memory.cljc index d31f577..0cf1383 100644 --- a/src/konserve/memory.cljc +++ b/src/konserve/memory.cljc @@ -1,5 +1,6 @@ (ns konserve.memory - "Address globally aggregated immutable key-value store(s)." + "Address globally aggregated immutable key-value store(s). + Does not support serialization." (:require [clojure.core.async :as async :refer [go ch [fd meta-size locked-cb _env] - (go - (let [[?err total-size] ( ch [fd meta-size blob _env] @@ -350,6 +374,7 @@ (.delete f) (try (sync-base parent-base) + nil (catch js/Error e e)))) @@ -498,19 +523,24 @@ (sync-base-async base)))) (defn detect-old-file-schema [& _args] (throw (js/Error "TODO detect-old-file-schema"))) -;; get-file-channel -;; migration (defn connect-fs-store "Create Filestore in given path. - Optional serializer, read-handlers, write-handlers, buffer-size and config (for fsync) can be changed. - Defaults are - {:base path - :serializer fressian-serializer - :read-handlers empty - :write-handlers empty - :buffer-size 1 MB - :config config} " + Optional serializer, read-handlers, write-handlers, buffer-size and config (for fsync) can be changed. + + + the `k/bget` callback gets different args depending on `:sync?` + - async bget callbacks recieve `{:input-stream }` akin to + the same call on the JVM filestore impl. These streams are opened to the + same fd that konserve is managing for the blob, so users should not call + destroy() or it will raise an error + - sync bget callbacks are called with `{:blob }` + + {:base path + :serializer fressian-serializer + :read-handlers empty + :write-handlers empty + :buffer-size 1 MB + :config config} " [path & {:keys [detect-old-file-schema? ephemeral? config] :or {detect-old-file-schema? false ephemeral? (fn [pathstr] diff --git a/src/konserve/utils.cljc b/src/konserve/utils.cljc index 9f783be..246928d 100644 --- a/src/konserve/utils.cljc +++ b/src/konserve/utils.cljc @@ -21,24 +21,23 @@ {:key key :type type :last-write (now)} (clojure.core/assoc old :last-write (now)))) -#?(:clj - (defmacro async+sync - [sync? async->sync async-code] - (let [async->sync (if (symbol? async->sync) - (or (resolve async->sync) - (when-let [_ns (or (get-in &env [:ns :use-macros async->sync]) - (get-in &env [:ns :uses async->sync]))] - (resolve (symbol (str _ns) (str async->sync))))) - async->sync)] - (assert (some? async->sync)) - `(if ~sync? - ~(clojure.walk/postwalk (fn [n] - (if-not (meta n) - (async->sync n n) ;; primitives have no metadata - (with-meta (async->sync n n) - (update (meta n) :tag (fn [t] (async->sync t t)))))) - async-code) - ~async-code)))) +(defmacro async+sync + [sync? async->sync async-code] + (let [async->sync (if (symbol? async->sync) + (or (resolve async->sync) + (when-let [_ns (or (get-in &env [:ns :use-macros async->sync]) + (get-in &env [:ns :uses async->sync]))] + (resolve (symbol (str _ns) (str async->sync))))) + async->sync)] + (assert (some? async->sync)) + `(if ~sync? + ~(clojure.walk/postwalk (fn [n] + (if-not (meta n) + (async->sync n n) ;; primitives have no metadata + (with-meta (async->sync n n) + (update (meta n) :tag (fn [t] (async->sync t t)))))) + async-code) + ~async-code))) (def ^:dynamic *default-sync-translation* '{go-try try diff --git a/test/konserve/cache_test.clj b/test/konserve/cache_test.clj deleted file mode 100644 index 8cb94c1..0000000 --- a/test/konserve/cache_test.clj +++ /dev/null @@ -1,67 +0,0 @@ -(ns konserve.cache-test - (:require [konserve.cache :as k] - [konserve.filestore :as fstore] - [clojure.core.async :refer [> list-keys (map #(clojure.core/dissoc % :last-write)) set) - true - (every? - (fn [{:keys [:last-write]}] - (= (type (java.util.Date.)) (type last-write))) - list-keys))) - - (doseq [to-delete [:baz :binbar :foolog]] - ( e :konserve.core/timestamp .getTime)]) (MemoryStore]] + [konserve.tests.cache :as ct] + [konserve.tests.gc :as gct])) + +(defn connect-mem-store + [init-atom & {:as params opts :opts}] + (let [store-config (merge {:state init-atom + :read-handlers (atom {}) + :write-handlers (atom {}) + :locks (atom {})} + (dissoc params :config)) + store (map->MemoryStore store-config)] + (if (:sync? opts) store (go store)))) + +#?(:clj + (deftest memory-store-compliance-test + (compliance-test (store-key key))] (testing "no lock, writes ok" @@ -168,3 +169,69 @@ (is (= [42 43] (MyRecord} + _ (MyRecord {:a 0 :b 1})] + (and + (is (nil? (MyRecord {:a 0 :b 1}) + res (and + (is [nil 42] (MyRecord {:a 0 :b 1}) + res (and + (is (= [nil my-record] (