Diagnosing changes to PrefectFuture handling in Prefect 3

[Andrew-S-Rosen]

I am trying to diagnose some changes to PrefectFuture in Prefect 3. Let me preface this by saying I know what I'm proposing is scandalous, but here it goes.

The following simplified toy example works as expected:

from prefect import task, flow

@task
def my_task():
    return 1

@flow
def my_subflow(dummy):
    return 1

@flow
def my_flow():
    result = my_task.submit()
    return my_subflow(result)

assert my_flow() == 1

However, the following runs into an infinite loop of get_item tasks before even getting to dispatching my_subflow even though there are no __getitem__ calls in the script. This behavior is unique to Prefect 3.

from prefect.futures import PrefectFuture
from prefect import flow, task

def _patched_getitem(self, index):
    @task
    def get_item(obj, key):
        return obj[key]

    return get_item.submit(self, index)

PrefectFuture.__getitem__ = _patched_getitem

@task
def my_task():
    return 1

@flow
def my_subflow(dummy):
    return 1

@flow
def my_flow():
    result = my_task.submit()
    return my_subflow(result)

assert my_flow() == 1

Does anyone on the maintainer team have an idea of why this might be? If you're curious, the reason I'm playing around with this monkeypatch at all is because of #10928, but that's not super relevant here.

The traceback shows:

20:51:15.823 | ERROR   | Task run 'get_item-0' - Task run failed with exception: TypeError("'int' object is not subscriptable") - Retries are exhausted

Traceback (most recent call last):
  File "/home/rosen/software/miniconda/envs/test/lib/python3.11/site-packages/prefect/task_engine.py", line 812, in run_context
    yield self
  File "/home/rosen/software/miniconda/envs/test/lib/python3.11/site-packages/prefect/task_engine.py", line 1435, in run_task_sync
    engine.call_task_fn(txn)
  File "/home/rosen/software/miniconda/envs/test/lib/python3.11/site-packages/prefect/task_engine.py", line 840, in call_task_fn
    result = call_with_parameters(self.task.fn, parameters)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rosen/software/miniconda/envs/test/lib/python3.11/site-packages/prefect/utilities/callables.py", line 208, in call_with_parameters
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "<ipython-input-5-dd76840fa2f2>", line 11, in get_item
    return obj[key]
           ~~~^^^^^
TypeError: 'int' object is not subscriptable

It's not clear to me why get_item-0 is even getting launched here.

[cicdw]

Hey @Andrew-S-Rosen -- I don't have a justification for why this is happening, but in the spirit of sharing info early I have found the line that triggers the getitem call from within the codebase here.

This encoder function is calling dict(result) on the result Future which triggers the infinite calls to __getitem__.

[Andrew-S-Rosen]

@cicdw: Thank you so, so much! That's it!! I am so relieved to put this mystery to rest. You are absolutely right!

Clearly, in the check to see if the result Future is serializable, Prefect was expecting an exception to be raised since PrefectFuture can't actually be converted to a dict. I hadn't realized that dict(obj) would trigger __getitem__ calls, and I got stuck in the try block forever. Well, that's a mystery solved.

Can I ask a practical question: how did you find out? Is the answer that I should start using a debugger like a good programmer? 😅

[cicdw]

Haha yea for sure, glad to help! I used pdbpp which is a beefed up / prettier version of pdb; I put a breakpoint() in the first line of _patched_getitem and then used u / d to go up and down the call stack to see where things originated.

[Andrew-S-Rosen]

Brilliant, thank you for the tip!