Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Show the Interceptor Pattern for Streaming Tables #28

Open
1 task
newfront opened this issue Apr 2, 2024 · 1 comment
Open
1 task

Show the Interceptor Pattern for Streaming Tables #28

newfront opened this issue Apr 2, 2024 · 1 comment

Comments

@newfront
Copy link
Owner

newfront commented Apr 2, 2024

Brought up by @AndreGodinho7 in the Delta User's slack channel.

When we are migrating from Table A (call it "last stable") and we need to move to the new "stable" table, while not "interrupting" the streams until we are "ready". If there is a new set of "columns" then those can get picked up by the streaming applications without problems. However, if there is a completely re-written table structure, and we need to use a new "uuid" - then the change is more complex. We want to shoot for less complicated.

  • Consider that there are live customers "streaming data customers"
  • You want to create a "new" table to "replace" the old "tableA"
  • You want to ensure there is a "backup plan" - so you intercept the "table" and "redirect"

This is not a simple established pattern. It is a full "table aliasing" concept.

  • talk about ways to get this to work (with Trino for example - you can "redirect" tables to new locations) - with managed tables and a table UUID that changes with "Delta" - we will need a more "seamless" means to handle the redirection.

  • Include an official writeup (blog post) to go along with the pattern.

@newfront
Copy link
Owner Author

newfront commented Apr 2, 2024

Evolution on https://en.wikipedia.org/wiki/Interceptor_pattern for streaming data tables.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant