Add Active Stack as part of configuration YAML

[strickvl]

Open Source Contributors Welcomed!

Please comment below if you would like to work on this issue!

Contact Details [Optional]

[email protected]

What happened?

After a discussion about the Run Config YAML and potential improvements, it became apparent that including the active stack name or ID in the configuration YAML would significantly enhance usability. This change would allow users to specify which stack the pipeline should run on directly within the config YAML.

Task Description

Update the ZenML configuration YAML structure to include an option for setting the active stack. Users should be able to specify the stack name or ID under a dedicated section in the YAML, which ZenML will then use to determine the active stack for pipeline execution.

Expected Outcome

• The configuration YAML should have a clear and user-friendly section for specifying the active stack.
• ZenML should correctly interpret and use the specified stack for running the pipeline.
• This update will provide users with greater control and flexibility in their pipeline execution environment.

Steps to Implement

• Design and implement a new section in the Run Config YAML format for specifying the active stack (name or ID).
• Update the ZenML backend to read and apply this configuration when initializing pipeline runs.
• Ensure backward compatibility and graceful handling when the stack section is not specified in existing YAMLs.
• Test the implementation thoroughly to ensure the correct stack is used based on the YAML configuration. (Include tests in PR).
• Update documentation to guide users on how to use this new feature in their YAML configurations.

Additional Context

This enhancement is part of ongoing efforts to make ZenML more flexible and user-centric, enabling users to tailor their MLOps workflows more precisely to their needs.

cc: @schustmi @znegrin

Code of Conduct

• I agree to follow this project's Code of Conduct

[kabinja]

Hello, if that task is still available, I can start working on it :)

[strickvl]

@kabinja yes for sure! let us know if you have any questions, and please do follow the CONTRIBUTING document, esp w/r/t to branching and making your PR off the develop branch!

[schustmi]

@kabinja One little detail for you that will be quite helpful: This is the pydantic class that defines the structure of the yaml configuration file.

[rahuja23]

Is this issue already resolved?

[htahir1]

@rahuja23 no we're still waiting for some activity here.. I think @kabinja seems to have moved on from the task. If not, happy to pick it up from here / make another PR for it

[htahir1]

There is a PR here that is stuck for the last 3 weeks #2370 (comment)

[rahuja23]

So I can contribute on this issue then?

[htahir1]

@rahuja23 yes i would say so, there is a PR already but its stale. You can start from there or from scratch

[rahuja23]

@htahir1 Perfect thanks!

[kabinja]

Hello, sorry for not getting back earlier. I won't have much time this summer so please go ahead. If you have some questions about what I tried to do, please do not hesitate to reach out to me.