[Core feature] Add support try and except in the workflow

[pingsutw]

Motivation: Why do you think this is important?

It's follow-up to these failure node PRs

• Add support failure node flytekit#840
• Add support failure node #4308

People might want to pass a specific task output to a failure node. To support that flytekit need to support try and except first, so users are be able to pass a promise to failure node.

Goal: What should the final outcome look like, ideally?

@workflow
def wf(name: str = "my_cluster"):
    id = create_cluster(name=name)
    try:
        t = t1(a=1, b="2")
    except:
        d = delete_cluster(id=id)  # failure node
    c >> t >> d

Describe alternatives you've considered

For now, failure node's input is same as workflow inputs.

@task(container_image=image_spec)
def clean_up(name: str):
    print(f"Deleting cluster {name}")

@workflow(on_failure=clean_up)
def wf2(name: str = "kevin"):
    c = create_cluster(name=name)
    t = t1(a=1, b="2")
    d = delete_cluster(name=name)
    c >> t >> d

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[dosubot]

🤖

Hello @pingsutw! I'm Dosu, your friendly neighborhood bot. I'm here to assist you with bugs, answer your questions, and guide you on your journey to becoming a contributor while we wait for a human maintainer. Delighted to meet you and looking forward to helping you out!

Thank you for your feature request. The idea of adding support for try and except in the workflow is indeed an interesting one and could potentially enhance the flexibility of error handling in Flyte workflows.

Currently, as you've pointed out, the failure node's input is the same as the workflow inputs. This is handled by the HandleFailureNodeOfSubWorkflow function in the subworkflowHandler struct, which is responsible for handling the failure of a subworkflow. If a failure node is defined in the subworkflow, it is executed and its status is handled accordingly. If no failure node is defined, the function directly returns a transition to the Failure phase with the original error. You can find more details about this in the subworkflow.go file.

As for the try and except functionality, it seems that it is not currently explicitly handled in the Flyte codebase. However, the concept of error handling is present in the form of node failure handling. When a node fails, the handleDownstream function is called to traverse downstream nodes. If an error occurs during this process, the function returns an undefined node status and the error. If no error occurs, the function returns a failed node status with the execution error of the node status. This is implemented in the executor.go file.

To implement the try and except functionality, one possible approach could be to extend the current node failure handling mechanism to support catching exceptions and executing alternative nodes (the except part). This would likely involve modifying the handleDownstream function and potentially other parts of the codebase.

However, this would be a significant change to the Flyte codebase and would require careful design and implementation to ensure that it does not introduce new issues or break existing functionality. It would also need to be thoroughly tested to ensure that it works correctly in all scenarios.

I hope this provides some insight into how this feature could potentially be implemented. If you have any further questions or need more information, please let me know.

Sources

• flytepropeller/pkg/controller/nodes/executor.go
• flytepropeller/pkg/controller/nodes/subworkflow/subworkflow.go

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[kumare3]

Adding try catch to workflow is not possible as they would need ast parsing in the dsl. Why not use eager mode?