Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

batch to store multiple key/value pairs. #74

Open
awb99 opened this issue Feb 26, 2022 · 1 comment
Open

batch to store multiple key/value pairs. #74

awb99 opened this issue Feb 26, 2022 · 1 comment

Comments

@awb99
Copy link

awb99 commented Feb 26, 2022

I tried to have this answered in the datahike slack chat, but nobody did know an answer.
I want to store 5000 values, each with a specific key in the konserve store.
From the documentation I am not able to determine how this works.
Currently I call (save [:invoice n] invoice-n) 5000 times. And this takes 30 minutes or so.
This is my save function:

(defn save [k item]
  (<!! (k/assoc-in store k item))
  )

How would I do it if I want to have konserve update it all in one go? Do I have to do this via core.async
primitive, or do I have to call (apply k/assoc-in) with multipe key/value pairs?

Thanks a lot.
In case there is documentation for this and I didn't find it, then I am sorry :-)

@whilo
Copy link
Member

whilo commented Aug 24, 2022

Konserve does not have a batch processing framework yet, e.g. map-reduce. Mostly it is used as a low-level storage layer interface for simple needs. Whenever you need to do selective but large update operations on its values or sophisticated queries it is beneficial to use Datahike or another index based data management solution, e.g. by rolling your own hitchhiker-tree or persistent sorted set index. What they basically do for you is to automatically balance the amount of read and write IO operations you have to do. In your case you might be able to put many more invoices than one into each konserve value and keep them in memory most of the time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants