Use BaseException instead of Exception to catch GeneratorExit failures

[SylvainCorlay]

Thanks to @maartenbreddels for taking the time to investigate this!

[SylvainCorlay]

Nice explanation about GeneratorExit exceptions when breaking loops in generators https://www.peterbe.com/plog/generatorexit.

[jasongrout]

Filling in one more detail from the docs: https://docs.python.org/3/library/exceptions.html#GeneratorExit - GeneratorExit inherits from BaseException "since it is technically not an error."

[SylvainCorlay]

I don't think this is actually the right fix to the issue... I am investigating more.

[jasongrout]

I was trying to figure out when you would get a GeneratorExit thrown in this case, but didn't get that far. Is that what is bothering you about this solution?

Since you do have a finally clause, you could also put the fallback into the finally (define output_cell above as None, and check to see if it is None in the finally and if it is, set it to the default). That's not figuring why there is an error, but it at least does make sure to define output_cell in the finally clause.

[maartenbreddels]

So the error doesn't happen due to us calling someone who fails (the execute_cell code path), but someone who calls us decides to bail out early, which might be an issue in a template, since that is the one that calls _jinja_cell_generator.

[jasongrout]

That makes more sense, thanks. To summarize, did the bailing happen because of the async problem, and while it was trying to bail, that's when we got the output cell error?

[jasongrout]

A further thought: according to the docs, a generator is not allowed to yield one more time on GeneratorExit. If it does, a RuntimeError is raised: https://docs.python.org/3/reference/expressions.html#generator.close

I think that catching GeneratorExit and yielding a value is the wrong thing to do here. Perhaps output_cell can be defined as None at the top, and the finally clause can check if the output_cell is not None, and if so, yield it, and change that BaseException back to Exception.

[jasongrout]

I think that catching GeneratorExit and yielding a value is the wrong thing to do here. Perhaps output_cell can be defined as None at the top, and the finally clause can check if the output_cell is not None, and if so, yield it, and change that BaseException back to Exception.

This still doesn't solve the problem with why the generator is exiting early, though?

[maartenbreddels]

No, I wonder if this improper handling hides the original exception. (from mobile phone)

…

On Fri, Jul 2, 2021, 18:15 Jason Grout ***@***.***> wrote: I think that catching GeneratorExit and yielding a value is the wrong thing to do here. Perhaps output_cell can be defined as None at the top, and the finally clause can check if the output_cell is not None, and if so, yield it, and change that BaseException back to Exception. This still doesn't solve the problem with *why* the generator is exiting early, though? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#919 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANPEPO6NIZIJJXNGFWP73DTVXQY3ANCNFSM47WHMKBQ> .

[jasongrout]

No, I wonder if this improper handling hides the original exception.

In the tracebacks I've seen, it has been something like what you posted about in erdewit/nest_asyncio#22, but it says that while it was handling that error, another error happened, which is this output_cell reference error. So I don't think it's masking the original exception, thanks to Python 3 exception chaining. However, there is still something more going on, maybe related to erdewit/nest_asyncio#22