Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor pipeline interface #901

Merged
merged 8 commits into from
Apr 5, 2024
Merged

Commits on Apr 4, 2024

  1. Rename pipeline package to dataset

    mrchtr authored and GeorgesLorre committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    85a44c7 View commit details
    Browse the repository at this point in the history
  2. Fix doc references

    GeorgesLorre committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    951a9ef View commit details
    Browse the repository at this point in the history
  3. Implementation new dataset interface (#902)

    First steps for the implementation of the new dataset interface: 
    - Removed Pipeline class
    - Added Workspace singleton to hold pipeline name, base_path, etc. ..
    (shouldn't be the focus of this PR)
    - Moved `Pipeline.read(..)` to Dataset class
    mrchtr authored and GeorgesLorre committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    722733c View commit details
    Browse the repository at this point in the history
  4. Initialise dataset from manifest (#911)

    Implements the new workflow including initialising datasets from
    manifest.
    
    - removes workspace again, using working_directory argument now to
    enable something like `fondant run ... --working_directory gs://...`
    - removes base_path from the manifest and changes the field location to
    absolute paths
    - changes the executor to generate absolute field locations
    - add `Dataset.read()` for initialising dataset from manifest path
    - adjust compiler code
    
    
    Try to create sketch out the flow and changes. Still have to fix the
    test cases and test the changes end to end for all runners.
    
    
    ![image](https://github.com/ml6team/fondant/assets/15777729/8c330daf-2076-493c-aea2-b7d3f85884b8)
    mrchtr authored and GeorgesLorre committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    9905be8 View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2024

  1. Configuration menu
    Copy the full SHA
    8c7ffbb View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    26a8218 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    dfba9f0 View commit details
    Browse the repository at this point in the history
  4. Feature/dataset based compilers (#917)

    Co-authored-by: Matthias Richter <[email protected]>
    GeorgesLorre and mrchtr committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    a6561be View commit details
    Browse the repository at this point in the history