Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add NoThrowDAG return type #29

Open
hannahilea opened this issue Sep 21, 2023 · 3 comments
Open

Add NoThrowDAG return type #29

hannahilea opened this issue Sep 21, 2023 · 3 comments

Comments

@hannahilea
Copy link
Contributor

Currently, NoThrowDag's are nice and simple; their return type is either the return type of their last step (if everything succeeds) or a NoThrowReturn{missing} if they fail.

In a private project, we've run into a case where---even on failure---it would be useful to have the intermediate results of upstream steps.

There are assorted ways this could be accomplished; one that could work (proposed by @kendal-s) is on failure to return a special type that the internal state of the results up to the point of failure, in addition to the basic NoThrowResult{missing} (which should still be returned, so that all warnings/errors are still captured.

I'd kind of like to keep the original basic implementation, but maybe we can add a special alternative function to transform!(dag, input) that's like transform_and_return_intermediates!(dag, input) or something, that always returns the intermediates (on success or failure) in addition to the original NoThrowResult? (I know use of "and" in function names is often a code smell; perhaps someone can come up with a better/alternative name.)

@hannahilea
Copy link
Contributor Author

On more thought, this should maybe instead be a kwarg to transform!... there are various other options we want to pass to a nothrowdag, and having a bunch of independent functions wouldn't let those compose nicely. (nor would a big old list of kwargs, but at least it would be useable!)

@ericphanson
Copy link
Member

What about a separate function like get_intermediates(dag) to allow getting whatever intermediates exist at call time? (Assuming they are stored in the dag; if not then this doesn’t make sense). Then one could query for them after running transform!.

@hannahilea
Copy link
Contributor Author

That would be a good idea, except that the dag is stateless and stores nothing. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants