Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support kafka sql snapshotting #4665

Merged
merged 13 commits into from
May 29, 2024

Conversation

carlesarnal
Copy link
Member

@carlesarnal carlesarnal commented May 10, 2024

This introduces initial support for kafkasql snapshotting.

This follows the below approach:

  • A call to the endpoint /admin/config/triggerSnapshot is made. For KafkaSql, this sends a snapshot marker message with a snapshot id to the journal topic, meaning that, at that point, a snapshot was triggered.
  • The consumer thread reads the message above and creates a sql dump of the H2 database, storing it in the configured location, and sends a message to a snapshots topic using the snapshotId above as the key of this message. This topic is used to keep track of all the snapshots.
  • At application startup, the snapshots topic is consumed by looking for the most recent snapshot. If there's one present, the application loads the sql dump retrieved from the location in the message and restores the internal database using it. This is done without actually calling any initialization process in the sql database. This is important for upgrade procedures.
  • Once the database has been restored, the consumer thread starts consuming all messages present in the journal topic, skipping all the messages until the corresponding snapshot marker message is found.
  • The rest of the messages on top of the snapshot are processed as normal and dispatched to the sql storage as appropriate.

One note:

  • The snapshot is currently triggered using an endpoint in the admin path that returns nothing, just the 200 if it's ok.

PS: Don't be scared by the 22k lines changed, 22,007 of them are just the snapshot I added for the unit test.

@carlesarnal carlesarnal force-pushed the support-kafka-sql-snapshotting branch from 332252b to 28e62a3 Compare May 21, 2024 07:28
@carlesarnal carlesarnal force-pushed the support-kafka-sql-snapshotting branch from 4d4475c to c6e71d3 Compare May 22, 2024 11:37
@carlesarnal carlesarnal force-pushed the support-kafka-sql-snapshotting branch from ad2d190 to 4ea84ba Compare May 28, 2024 13:04
@carlesarnal
Copy link
Member Author

@EricWittmann this is good to go now.

@carlesarnal carlesarnal linked an issue May 29, 2024 that may be closed by this pull request
@EricWittmann EricWittmann merged commit 2ab68fa into Apicurio:main May 29, 2024
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants