Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make TTL configurable via config and reconcile it #123

Open
leoluk opened this issue Apr 17, 2023 · 1 comment
Open

Make TTL configurable via config and reconcile it #123

leoluk opened this issue Apr 17, 2023 · 1 comment

Comments

@leoluk
Copy link
Member

leoluk commented Apr 17, 2023

We currently default to 1 week and tell users to customize it using alter table default.flows_raw modify ttl Date + toIntervalDay(20).

Now that we auto-reconcile, we should make this a config item and have the reconciler apply it automatically (+making sure it gets applied to old partitions, maybe CH even does this automatically now).

@fionera
Copy link
Collaborator

fionera commented Jun 10, 2023

This is harder than it seems as we have to prevent users to accidentally nuke all their data. My current Idea is to move all operations to the reconciler including the creation of flows_raw but still keep migrations around for destruction of things. Also if the reconciler wants to reduce the ttl, delete anything etc. it should cancel and throw a warning.

Maybe we can create a table where we write log messages into and display them inside grafana to not have the user run in the dark?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants