Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scheduled cache priming: unbounded znode version? #120

Open
soxofaan opened this issue Sep 28, 2023 · 1 comment
Open

Scheduled cache priming: unbounded znode version? #120

soxofaan opened this issue Sep 28, 2023 · 1 comment

Comments

@soxofaan
Copy link
Member

soxofaan commented Sep 28, 2023

Follow up of #74:
there is now a X-minutely cron background task that primes the caches (regardless of existing values). Consequently the Zookeeper znode version is bumped roughly 100 times a day (even though the actual data does not change that often).
Is there a risk of letting this grow unboundedly?

Some examples of current state after running this for just a couple of days (prod cache):

  • version 1193 /openeo/aggregator/cache/CollectionCatalog/all
  • version 1023 /openeo/aggregator/cache/Processing/all/1.0.0
  • version 944 /openeo/aggregator/cache/general/file_formats
@soxofaan
Copy link
Member Author

soxofaan commented Sep 28, 2023

possible solutions:

  • just prune the whole /openeo/aggregator/cache tree every X months. If I understand correctly, version start over from 0 again
  • switch to a different caching solution (zookeeper is not really a caching solution anyway)
  • make "cache priming" implementation smarter to not bump version if it is not necessary (new value equals cached version and mtime is not too old yet)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant