-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor pipeline interface #901
Merged
Merged
Commits on Apr 4, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 85a44c7 - Browse repository at this point
Copy the full SHA 85a44c7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 951a9ef - Browse repository at this point
Copy the full SHA 951a9efView commit details -
Implementation new dataset interface (#902)
First steps for the implementation of the new dataset interface: - Removed Pipeline class - Added Workspace singleton to hold pipeline name, base_path, etc. .. (shouldn't be the focus of this PR) - Moved `Pipeline.read(..)` to Dataset class
Configuration menu - View commit details
-
Copy full SHA for 722733c - Browse repository at this point
Copy the full SHA 722733cView commit details -
Initialise dataset from manifest (#911)
Implements the new workflow including initialising datasets from manifest. - removes workspace again, using working_directory argument now to enable something like `fondant run ... --working_directory gs://...` - removes base_path from the manifest and changes the field location to absolute paths - changes the executor to generate absolute field locations - add `Dataset.read()` for initialising dataset from manifest path - adjust compiler code Try to create sketch out the flow and changes. Still have to fix the test cases and test the changes end to end for all runners. ![image](https://github.com/ml6team/fondant/assets/15777729/8c330daf-2076-493c-aea2-b7d3f85884b8)
Configuration menu - View commit details
-
Copy full SHA for 9905be8 - Browse repository at this point
Copy the full SHA 9905be8View commit details
Commits on Apr 5, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 8c7ffbb - Browse repository at this point
Copy the full SHA 8c7ffbbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 26a8218 - Browse repository at this point
Copy the full SHA 26a8218View commit details -
Configuration menu - View commit details
-
Copy full SHA for dfba9f0 - Browse repository at this point
Copy the full SHA dfba9f0View commit details -
Feature/dataset based compilers (#917)
Co-authored-by: Matthias Richter <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for a6561be - Browse repository at this point
Copy the full SHA a6561beView commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.