Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

InterpreterPoolExecutor Raises Attribute Error on Lambdas #125865

Open
paultiq opened this issue Oct 23, 2024 · 3 comments
Open

InterpreterPoolExecutor Raises Attribute Error on Lambdas #125865

paultiq opened this issue Oct 23, 2024 · 3 comments
Labels
pending The issue will be closed if no feedback is provided topic-subinterpreters type-bug An unexpected behavior, bug, or error

Comments

@paultiq
Copy link

paultiq commented Oct 23, 2024

Bug report

Bug description:

3.14.0a1+ 34653bb, Windows 64bit

The following code raises an attribute error:

from concurrent.futures.interpreter import InterpreterPoolExecutor

with InterpreterPoolExecutor() as executor:
    future = executor.submit(lambda x: 10)
    
result = future.result() 
print(result)

This code runs:

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor() as executor:
    future = executor.submit(lambda: 10)

result = future.result()  # Get the result of the function
print(result)

Error:

AttributeError: module '__main__' has no attribute '<lambda>'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\scratch\cpython\PCbuild\amd64\test1.py", line 4, in <module>
    future = executor.submit(lambda: 10)
  File "C:\scratch\cpython\Lib\concurrent\futures\thread.py", line 210, in submit
    task = self._resolve_work_item_task(fn, args, kwargs)
  File "C:\scratch\cpython\Lib\concurrent\futures\interpreter.py", line 64, in resolve_task
    data = pickle.dumps((fn, args, kwargs))
_pickle.PicklingError: Can't pickle <function <lambda> at 0x000001455882EDA0>: it's not found as __main__.<lambda>
when serializing tuple item 0

CPython versions tested on:

CPython main branch

Operating systems tested on:

Windows

@paultiq paultiq added the type-bug An unexpected behavior, bug, or error label Oct 23, 2024
@paultiq
Copy link
Author

paultiq commented Oct 23, 2024

Unclear if related to #125864, so I filed separately, but they're similar.

@ZeroIntensity ZeroIntensity added the pending The issue will be closed if no feedback is provided label Oct 23, 2024
@ZeroIntensity
Copy link
Member

Yeah, lambdas can't get picked. I don't think there's anything that can be done.

@paultiq
Copy link
Author

paultiq commented Oct 23, 2024

From the doc, https://github.com/python/cpython/blob/main/Doc/library/concurrent.futures.rst, I expected InterpreterPoolExecutor to behave similarly to ThreadPoolExecutor... yet in this regards, it behaves similar to ProcessPoolExecutor.

The opening paragraph says: "It is a :class:ThreadPoolExecutor subclass, which means each worker is running in its own thread.", which perhaps could be clarified that certain behaviors are more consistent with ProcessPoolExecutor.

** In short: the docs led me to expect that code that runs on a ThreadPoolExecutor would, generally speaking, work in a InterpreterPoolExecutor... a clearer comment up top that this isn't the case may avoid some user errors like this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pending The issue will be closed if no feedback is provided topic-subinterpreters type-bug An unexpected behavior, bug, or error
Projects
Status: Todo
Development

No branches or pull requests

3 participants