Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dump server metadata to top of processed files #11

Open
junkert opened this issue Mar 12, 2019 · 1 comment
Open

Add dump server metadata to top of processed files #11

junkert opened this issue Mar 12, 2019 · 1 comment
Labels
enhancement New feature or request

Comments

@junkert
Copy link
Collaborator

junkert commented Mar 12, 2019

@wavemoran brought up a good point today during a quick sprint demo of the project.

What would happen if we swapped the loader and processor server information in the configuration between the processing stage and loading stage (same as dump, processor, and loader all having identical server, database, and credentials set)? In this case we would be taking a dump file from the same server we would be loading into (same database, schemas, allthethings). This would cause issues if the dump and load servers were both production servers 😱... we do, however, make sure to NEVER run a DROP DATABASE during any execution paths in the load command pathways. Instead we do an atomic renaming of the primary database we are loading into by adding a timestamp to the name. Since we rename the primary database to the timestamp name we can then rename the anonymized database to the primary name.

We can fix this issue by adding metadata such as: hostname, port, and database to the pre-processor. This way we can read this data during the load command and verify that the server we are loading into is not the same server we dumped from.

@junkert junkert added the enhancement New feature or request label Mar 12, 2019
@junkert
Copy link
Collaborator Author

junkert commented Mar 12, 2019

we should also add the ability to --disable this check just incase someone else has a use case where the ability to dump/load from the same resource is valid. By default we should not allow this behavior.

@junkert junkert changed the title Add checksumming to dump and processed files Add dump server metadata to top of processed files Mar 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant