Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apply recipe directly on archived dataset #123

Open
heikomuller opened this issue Apr 28, 2021 · 0 comments
Open

Apply recipe directly on archived dataset #123

heikomuller opened this issue Apr 28, 2021 · 0 comments

Comments

@heikomuller
Copy link
Member

Is your feature request related to a problem? Please describe.
When applying the recipe operations for a sample dataset on the full dataset we currently need to checkout the previous snapshot, modify it and write it back to the archive. This is not only inefficient but can also cause problems for large datasets that do not fit into main memory.

Describe the solution you'd like
The latest version of histore supports applying modification operations directly on the archive (using the DatasetOperator). We should make use of this feature in openclean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant