Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Periodically run the syncGraphersToR2 devtool to avoid R2 configs drifting via DB ones after migrations #4028

Open
danyx23 opened this issue Oct 7, 2024 · 0 comments

Comments

@danyx23
Copy link
Contributor

danyx23 commented Oct 7, 2024

With #3827 we introduced a sync mechanism that stores the full grapher configs in R2 so that we can fetch them via a simple API instead of extracting this information from HTML like we did before.

These R2 files are updated whenever a chart is saved via the admin rest API. This covers all cases of normal config write scenarios (e.g. inside the admin, the ETL updates, ...) but it leaves out one point of modification: direct database changes, especially migrations.

We don't change chart configs in our db via migrations all the time but in the average year we probably do this 2-4 times. We could either:

  • manually call the syncGraphersToR2 script
  • try to run in after migrations run in the correct context
  • just run it periodically (e.g. every 10 min).

A single invocation if all charts are up to date only takes ~10 seconds so running the sync script is not particularly expensive but we should probably ensure that only of these processes runs at any given time (probably best via buildkite)

@danyx23 danyx23 changed the title Periodically run the syncGraphersToR2 devtool to avoid drifting after migrations Periodically run the syncGraphersToR2 devtool to avoid R2 configs drifting via DB ones after migrations Oct 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant