Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking test_gil_scoped.py DEADLOCKs #4373

Open
rwgk opened this issue Dec 1, 2022 · 8 comments
Open

Tracking test_gil_scoped.py DEADLOCKs #4373

rwgk opened this issue Dec 1, 2022 · 8 comments

Comments

@rwgk
Copy link
Collaborator

rwgk commented Dec 1, 2022

Using this Issue to track test_gil_scoped.py failures like this one:

E               RuntimeError: DEADLOCK, most likely, exactly what this test is meant to detect.

Background: PRs #4105, #4306

@rwgk rwgk added triage New bug, unverified and removed triage New bug, unverified labels Dec 1, 2022
@rwgk
Copy link
Collaborator Author

rwgk commented Dec 1, 2022

PR #4371

https://github.com/pybind/pybind11/actions/runs/3590629225/jobs/6044221872

CI — 🐍 3.6 • macos-latest • x64

platform darwin -- Python 3.6.15, pytest-7.0.0, pluggy-1.0.0
C++ Info: Apple LLVM 14.0.0 (clang-1400.0.29.102) C++11 __pybind11_internals_v4_clang_libcpp_cxxabi1002_sh_def__ PYBIND11_SIMPLE_GIL_MANAGEMENT=True
...
=================================== FAILURES ===================================
_______________ test_run_in_process_direct[test_nested_acquire] ________________

test_fn = <function test_nested_acquire at 0x104218e18>

    @pytest.mark.parametrize("test_fn", ALL_BASIC_TESTS_PLUS_INTENTIONAL_DEADLOCK)
    def test_run_in_process_direct(test_fn):
        """Makes sure there is no GIL deadlock when using processes.
    
        This test is for completion, but it was never an issue.
        """
>       assert _run_in_process(test_fn) == 0

test_fn    = <function test_nested_acquire at 0x104218e18>

test_gil_scoped.py:244: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

target = <function test_nested_acquire at 0x104218e18>, args = (), kwargs = {}
test_fn = <function test_nested_acquire at 0x104218e18>, timeout = 10
process = <Process(Process-70, stopped[255] daemon)>
t_start = 1669882422.6540549, t_delta = 10.150235176086426, @py_assert1 = None

    def _run_in_process(target, *args, **kwargs):
        if len(args) == 0:
            test_fn = target
        else:
            test_fn = args[0]
        # Do not need to wait much, 10s should be more than enough.
        timeout = 0.1 if test_fn is _intentional_deadlock else 10
        process = multiprocessing.Process(target=target, args=args, kwargs=kwargs)
        process.daemon = True
        try:
            t_start = time.time()
            process.start()
            if timeout >= [100](https://github.com/pybind/pybind11/actions/runs/3590629225/jobs/6044221872#step:12:101):  # For debugging.
                print(
                    "\nprocess.pid STARTED", process.pid, (sys.argv, target, args, kwargs)
                )
                print(f"COPY-PASTE-THIS: gdb {sys.argv[0]} -p {process.pid}", flush=True)
            process.join(timeout=timeout)
            if timeout >= 100:
                print("\nprocess.pid JOINED", process.pid, flush=True)
            t_delta = time.time() - t_start
            if process.exitcode == 66 and m.defined_THREAD_SANITIZER:  # Issue #2754
                # WOULD-BE-NICE-TO-HAVE: Check that the message below is actually in the output.
                # Maybe this could work:
                # https://gist.github.com/alexeygrigorev/01ce847f2e721b513b42ea4a6c96905e
                pytest.skip(
                    "ThreadSanitizer: starting new threads after multi-threaded fork is not supported."
                )
            elif test_fn is _intentional_deadlock:
                assert process.exitcode is None
                return 0
            elif process.exitcode is None:
                assert t_delta > 0.9 * timeout
                msg = "DEADLOCK, most likely, exactly what this test is meant to detect."
                if env.PYPY and env.WIN:
                    pytest.skip(msg)
>               raise RuntimeError(msg)
E               RuntimeError: DEADLOCK, most likely, exactly what this test is meant to detect.

args       = ()
kwargs     = {}
msg        = 'DEADLOCK, most likely, exactly what this test is meant to detect.'
process    = <Process(Process-70, stopped[255] daemon)>
t_delta    = 10.150235176086426
t_start    = 1669882422.6540549
target     = <function test_nested_acquire at 0x[104](https://github.com/pybind/pybind11/actions/runs/3590629225/jobs/6044221872#step:12:105)218e18>
test_fn    = <function test_nested_acquire at 0x104218e18>
timeout    = 10

test_gil_scoped.py:186: RuntimeError

@rwgk
Copy link
Collaborator Author

rwgk commented Dec 4, 2022

PR #4382

https://github.com/pybind/pybind11/actions/runs/3615276319/jobs/6092282270

🐍 pypy-3.9 • macos-latest • x64

platform darwin -- Python 3.9.12[pypy-7.3.9-final], pytest-7.0.0, pluggy-1.0.0
C++ Info: Apple LLVM 14.0.0 (clang-1400.0.29.102) C++11 __pybind11_internals_v4_clang_libcpp_cxxabi1002__ PYBIND11_SIMPLE_GIL_MANAGEMENT=True
...
test_gil_scoped.py ..................................................... [ 46%]
.......F.............................                                    [ 53%]
test_iostream.py ......................                                  [ 57%]
test_kwargs_and_defaults.py ........                                     [ 58%]
test_local_bindings.py ..........                                        [ 60%]
test_methods_and_attributes.py ............xx........                    [ 64%]
        process.daemon = True
        try:
            t_start = time.time()
            process.start()
            if timeout >= 100:  # For debugging.
                print(
                    "\nprocess.pid STARTED", process.pid, (sys.argv, target, args, kwargs)
                )
                print(f"COPY-PASTE-THIS: gdb {sys.argv[0]} -p {process.pid}", flush=True)
            process.join(timeout=timeout)
            if timeout >= 100:
                print("\nprocess.pid JOINED", process.pid, flush=True)
            t_delta = time.time() - t_start
            if process.exitcode == 66 and m.defined_THREAD_SANITIZER:  # Issue #2754
                # WOULD-BE-NICE-TO-HAVE: Check that the message below is actually in the output.
                # Maybe this could work:
                # https://gist.github.com/alexeygrigorev/01ce847f2e721b513b42ea4a6c96905e
                pytest.skip(
                    "ThreadSanitizer: starting new threads after multi-threaded fork is not supported."
                )
            elif test_fn is _intentional_deadlock:
                assert process.exitcode is None
                return 0
            elif process.exitcode is None:
                assert t_delta > 0.9 * timeout
                msg = "DEADLOCK, most likely, exactly what this test is meant to detect."
                if env.PYPY and env.WIN:
                    pytest.skip(msg)
>               raise RuntimeError(msg)
E               RuntimeError: DEADLOCK, most likely, exactly what this test is meant to detect.

args       = (<function test_cross_module_gil_inner_custom_released at 0x00007fe8111f8ac0>,)
kwargs     = {'num_threads': 8, 'parallel': False}
msg        = 'DEADLOCK, most likely, exactly what this test is meant to detect.'
process    = <Process name='Process-43' pid=9763 parent=8413 stopped exitcode=-SIGTERM daemon>
t_delta    = 10.249207973480225
t_start    = 1670[19](https://github.com/pybind/pybind11/actions/runs/3615276319/jobs/6092282270#step:12:20)1997.6[23](https://github.com/pybind/pybind11/actions/runs/3615276319/jobs/6092282270#step:12:24)723
target     = <function _run_in_threads at 0x00007fe8109294c0>
test_fn    = <function test_cross_module_gil_inner_custom_released at 0x00007fe8111f8ac0>
timeout    = 10

test_gil_scoped.py:186: RuntimeError

@rwgk
Copy link
Collaborator Author

rwgk commented Dec 5, 2022

PR #4384

https://github.com/pybind/pybind11/actions/runs/3618069828/jobs/6097514565

🐍 3.6 • macos-latest • x64

platform darwin -- Python 3.6.15, pytest-7.0.0, pluggy-1.0.0
C++ Info: Apple LLVM 14.0.0 (clang-1400.0.29.102) C++17 __pybind11_internals_v4_clang_libcpp_cxxabi1002__ PYBIND11_SIMPLE_GIL_MANAGEMENT=False
rootdir: /Users/runner/work/pybind11/pybind11/tests, configfile: pytest.ini
plugins: timeout-2.1.0
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 790 items

../../tests/test_async.py ..                                             [  0%]
../../tests/test_buffers.py .........                                    [  1%]
../../tests/test_builtin_casters.py ....................                 [  3%]
../../tests/test_call_policies.py ........                               [  4%]
../../tests/test_callbacks.py .............s                             [  6%]
../../tests/test_chrono.py ...........................................   [ 12%]
../../tests/test_class.py ...............................                [ 16%]
../../tests/test_const_name.py ......................                    [ 18%]
../../tests/test_constants_and_functions.py .....                        [ [19](https://github.com/pybind/pybind11/actions/runs/3618069828/jobs/6097514565#step:18:20)%]
../../tests/test_copy_move.py ........                                   [ [20](https://github.com/pybind/pybind11/actions/runs/3618069828/jobs/6097514565#step:18:21)%]
../../tests/test_custom_type_casters.py ...                              [ 20%]
../../tests/test_custom_type_setup.py ..                                 [ [21](https://github.com/pybind/pybind11/actions/runs/3618069828/jobs/6097514565#step:18:22)%]
../../tests/test_docstring_options.py .                                  [ 21%]
../../tests/test_eigen_matrix.py ..............................          [ 25%]
../../tests/test_eigen_tensor.py ....................................... [ 30%]
......................................................................   [ 38%]
../../tests/test_enum.py .........                                       [ 40%]
../../tests/test_eval.py ....                                            [ 40%]
../../tests/test_exceptions.py ........................                  [ 43%]
../../tests/test_factory_constructors.py ...............                 [ 45%]
../../tests/test_gil_scoped.py ........................................F [ 50%]
.................................................                        [ 56%]
../../tests/test_iostream.py ......................                      [ 59%]
            test_fn = target
        else:
            test_fn = args[0]
        # Do not need to wait much, 10s should be more than enough.
        timeout = 0.1 if test_fn is _intentional_deadlock else 10
        process = multiprocessing.Process(target=target, args=args, kwargs=kwargs)
        process.daemon = True
        try:
            t_start = time.time()
            process.start()
            if timeout >= 100:  # For debugging.
                print(
                    "\nprocess.pid STARTED", process.pid, (sys.argv, target, args, kwargs)
                )
                print(f"COPY-PASTE-THIS: gdb {sys.argv[0]} -p {process.pid}", flush=True)
            process.join(timeout=timeout)
            if timeout >= 100:
                print("\nprocess.pid JOINED", process.pid, flush=True)
            t_delta = time.time() - t_start
            if process.exitcode == 66 and m.defined_THREAD_SANITIZER:  # Issue #2754
                # WOULD-BE-NICE-TO-HAVE: Check that the message below is actually in the output.
                # Maybe this could work:
                # https://gist.github.com/alexeygrigorev/01ce847f2e721b513b42ea4a6c96905e
                pytest.skip(
                    "ThreadSanitizer: starting new threads after multi-threaded fork is not supported."
                )
            elif test_fn is _intentional_deadlock:
                assert process.exitcode is None
                return 0
            elif process.exitcode is None:
                assert t_delta > 0.9 * timeout
                msg = "DEADLOCK, most likely, exactly what this test is meant to detect."
                if env.PYPY and env.WIN:
                    pytest.skip(msg)
>               raise RuntimeError(msg)
E               RuntimeError: DEADLOCK, most likely, exactly what this test is meant to detect.

args       = (<function test_cross_module_gil_released at 0x103a96e18>,)
kwargs     = {'num_threads': 8, 'parallel': True}
msg        = 'DEADLOCK, most likely, exactly what this test is meant to detect.'
process    = <Process(Process-23, stopped[255] daemon)>
t_delta    = 10.005295753479004
t_start    = 1670[22](https://github.com/pybind/pybind11/actions/runs/3618069828/jobs/6097514565#step:18:23)8304.4042292
target     = <function _run_in_threads at 0x103a3c730>
test_fn    = <function test_cross_module_gil_released at 0x103a96e18>
timeout    = 10

../../tests/test_gil_scoped.py:186: RuntimeError
=========================== short test summary info ============================
SKIPPED [1] ../../tests/test_callbacks.py:203: Current PYBIND11_INTERNALS_VERSION too low
SKIPPED [1] ../../tests/test_pytypes.py:397: Not defined: PYBIND11_HANDLE_REF_DEBUG
SKIPPED [1] ../../tests/test_stl.py:143: no <experimental/optional>
FAILED ../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[test_cross_module_gil_released]
================== 1 failed, 786 passed, 3 skipped in 59.09s ===================

@rwgk
Copy link
Collaborator Author

rwgk commented Dec 9, 2022

PR #2768

https://github.com/pybind/pybind11/actions/runs/3655052690/jobs/6176034826

🐍 pypy-3.7 • macos-latest • x64

============================= test session starts ==============================
platform darwin -- Python 3.7.13[pypy-7.3.9-final], pytest-7.0.0, pluggy-1.0.0
C++ Info: Apple LLVM 14.0.0 (clang-1400.0.29.102) C++17 __pybind11_internals_v10000000_clang_libcpp_cxxabi1002__ PYBIND11_SIMPLE_GIL_MANAGEMENT=True
rootdir: /Users/runner/work/pybind11/pybind11/tests, configfile: pytest.ini
plugins: timeout-2.1.0
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 100 items

../../tests/test_call_policies.py X.x.....                               [  8%]
../../tests/test_gil_scoped.py ......................................... [ 49%]
...............................................F.                        [ 98%]
../../tests/test_thread.py ..                                            [100%]

=================================== FAILURES ===================================
_____ test_run_in_process_direct[test_multi_acquire_release_cross_module] ______

test_fn = <function test_multi_acquire_release_cross_module at 0x00007fac9fcc9380>

    @pytest.mark.parametrize("test_fn", ALL_BASIC_TESTS_PLUS_INTENTIONAL_DEADLOCK)
    def test_run_in_process_direct(test_fn):
        """Makes sure there is no GIL deadlock when using processes.
    
        This test is for completion, but it was never an issue.
        """
>       assert _run_in_process(test_fn) == 0

test_fn    = <function test_multi_acquire_release_cross_module at 0x00007fac9fcc9380>

../../tests/test_gil_scoped.py:244: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

target = <function test_multi_acquire_release_cross_module at 0x00007fac9fcc9380>
args = (), kwargs = {}
test_fn = <function test_multi_acquire_release_cross_module at 0x00007fac9fcc9380>
timeout = 10, process = <Process(Process-71, stopped[SIGTERM] daemon)>
t_start = [16](https://github.com/pybind/pybind11/actions/runs/3655052690/jobs/6176034826#step:22:17)70568874.939685, t_delta = 10.020744800567627, @py_assert1 = None

    def _run_in_process(target, *args, **kwargs):
        if len(args) == 0:
            test_fn = target
        else:
            test_fn = args[0]
        # Do not need to wait much, 10s should be more than enough.
        timeout = 0.1 if test_fn is _intentional_deadlock else 10
        process = multiprocessing.Process(target=target, args=args, kwargs=kwargs)
        process.daemon = True
        try:
            t_start = time.time()
            process.start()
            if timeout >= 100:  # For debugging.
                print(
                    "\nprocess.pid STARTED", process.pid, (sys.argv, target, args, kwargs)
                )
                print(f"COPY-PASTE-THIS: gdb {sys.argv[0]} -p {process.pid}", flush=True)
            process.join(timeout=timeout)
            if timeout >= 100:
                print("\nprocess.pid JOINED", process.pid, flush=True)
            t_delta = time.time() - t_start
            if process.exitcode == 66 and m.defined_THREAD_SANITIZER:  # Issue #2754
                # WOULD-BE-NICE-TO-HAVE: Check that the message below is actually in the output.
                # Maybe this could work:
                # https://gist.github.com/alexeygrigorev/01ce847f2e721b513b42ea4a6c96905e
                pytest.skip(
                    "ThreadSanitizer: starting new threads after multi-threaded fork is not supported."
                )
            elif test_fn is _intentional_deadlock:
                assert process.exitcode is None
                return 0
            elif process.exitcode is None:
                assert t_delta > 0.9 * timeout
                msg = "DEADLOCK, most likely, exactly what this test is meant to detect."
                if env.PYPY and env.WIN:
                    pytest.skip(msg)
>               raise RuntimeError(msg)
E               RuntimeError: DEADLOCK, most likely, exactly what this test is meant to detect.

args       = ()
kwargs     = {}
msg        = 'DEADLOCK, most likely, exactly what this test is meant to detect.'
process    = <Process(Process-71, stopped[SIGTERM] daemon)>
t_delta    = 10.020744800567627
t_start    = 1670568874.939685
target     = <function test_multi_acquire_release_cross_module at 0x00007fac9fcc9380>
test_fn    = <function test_multi_acquire_release_cross_module at 0x00007fac9fcc9380>
timeout    = 10

../../tests/test_gil_scoped.py:[18](https://github.com/pybind/pybind11/actions/runs/3655052690/jobs/6176034826#step:22:19)6: RuntimeError

@rwgk
Copy link
Collaborator Author

rwgk commented Jan 24, 2023

PR #4463

https://github.com/pybind/pybind11/actions/runs/3993013804/jobs/6849373017

🐍 3.10 • macos-latest • x64

============================= test session starts ==============================
platform darwin -- Python 3.10.9, pytest-7.0.0, pluggy-1.0.0
C++ Info: Apple LLVM 14.0.0 (clang-1400.0.29.202) C++11 __pybind11_internals_v4_clang_libcpp_cxxabi1002__ PYBIND11_SIMPLE_GIL_MANAGEMENT=True
...
=================================== FAILURES ===================================
_____ test_run_in_process_multiple_threads_sequential[test_nested_acquire] _____

test_fn = <function test_nested_acquire at 0x10ea3d480>

    @pytest.mark.parametrize("test_fn", ALL_BASIC_TESTS_PLUS_INTENTIONAL_DEADLOCK)
    def test_run_in_process_multiple_threads_sequential(test_fn):
        """Makes sure there is no GIL deadlock when running in a thread multiple times sequentially.
    
        It runs in a separate process to be able to stop and assert if it deadlocks.
        """
>       assert _run_in_process(_run_in_threads, test_fn, num_threads=8, parallel=False) == 0

test_fn    = <function test_nested_acquire at 0x10ea3d480>

test_gil_scoped.py:234: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

target = <function _run_in_threads at 0x10ea3d750>
args = (<function test_nested_acquire at 0x10ea3d480>,)
kwargs = {'num_threads': 8, 'parallel': False}
test_fn = <function test_nested_acquire at 0x10ea3d480>, timeout = 10
process = <Process name='Process-52' pid=17490 parent=16338 stopped exitcode=0 daemon>
t_start = 1674534973.682475, t_delta = 10.0825610[160](https://github.com/pybind/pybind11/actions/runs/3993013804/jobs/6849373017#step:12:161)82764, @py_assert1 = None

    def _run_in_process(target, *args, **kwargs):
        if len(args) == 0:
            test_fn = target
        else:
            test_fn = args[0]
        # Do not need to wait much, 10s should be more than enough.
        timeout = 0.1 if test_fn is _intentional_deadlock else 10
        process = multiprocessing.Process(target=target, args=args, kwargs=kwargs)
        process.daemon = True
        try:
            t_start = time.time()
            process.start()
            if timeout >= 100:  # For debugging.
                print(
                    "\nprocess.pid STARTED", process.pid, (sys.argv, target, args, kwargs)
                )
                print(f"COPY-PASTE-THIS: gdb {sys.argv[0]} -p {process.pid}", flush=True)
            process.join(timeout=timeout)
            if timeout >= 100:
                print("\nprocess.pid JOINED", process.pid, flush=True)
            t_delta = time.time() - t_start
            if process.exitcode == 66 and m.defined_THREAD_SANITIZER:  # Issue #2754
                # WOULD-BE-NICE-TO-HAVE: Check that the message below is actually in the output.
                # Maybe this could work:
                # https://gist.github.com/alexeygrigorev/01ce847f2e721b513b42ea4a6c96905e
                pytest.skip(
                    "ThreadSanitizer: starting new threads after multi-threaded fork is not supported."
                )
            elif test_fn is _intentional_deadlock:
                assert process.exitcode is None
                return 0
            elif process.exitcode is None:
                assert t_delta > 0.9 * timeout
                msg = "DEADLOCK, most likely, exactly what this test is meant to detect."
                if env.PYPY and env.WIN:
                    pytest.skip(msg)
>               raise RuntimeError(msg)
E               RuntimeError: DEADLOCK, most likely, exactly what this test is meant to detect.

args       = (<function test_nested_acquire at 0x10ea3d480>,)
kwargs     = {'num_threads': 8, 'parallel': False}
msg        = 'DEADLOCK, most likely, exactly what this test is meant to detect.'
process    = <Process name='Process-52' pid=17490 parent=[163](https://github.com/pybind/pybind11/actions/runs/3993013804/jobs/6849373017#step:12:164)38 stopped exitcode=0 daemon>
t_delta    = 10.082561016082764
t_start    = [167](https://github.com/pybind/pybind11/actions/runs/3993013804/jobs/6849373017#step:12:168)4534973.682475
target     = <function _run_in_threads at 0x10ea3d750>
test_fn    = <function test_nested_acquire at 0x10ea3d480>
timeout    = 10

test_gil_scoped.py:[186](https://github.com/pybind/pybind11/actions/runs/3993013804/jobs/6849373017#step:12:187): RuntimeError
=========================== short test summary info ============================
SKIPPED [1] test_builtin_casters.py:145: no <string_view>
SKIPPED [1] test_callbacks.py:[213](https://github.com/pybind/pybind11/actions/runs/3993013804/jobs/6849373017#step:12:214): Current PYBIND11_INTERNALS_VERSION too low
SKIPPED [1] test_copy_move.py:77: no <optional>
SKIPPED [1] test_pytypes.py:403: Not defined: PYBIND11_HANDLE_REF_DEBUG
SKIPPED [1] test_sequences_and_iterators.py:13: no <optional>
SKIPPED [1] test_stl.py:111: no <optional>
SKIPPED [1] test_stl.py:143: no <experimental/optional>
SKIPPED [1] test_stl.py:234: no <filesystem>
SKIPPED [1] test_stl.py:270: no std::monostate
FAILED test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_nested_acquire]
============= 1 failed, 784 passed, 9 skipped in 71.28s (0:01:11) ==============

@rwgk
Copy link
Collaborator Author

rwgk commented Feb 8, 2023

PR #4504

https://github.com/pybind/pybind11/actions/runs/4121981206/jobs/7119639395

🐍 pypy-3.8 • macos-latest • x64

============================= test session starts ==============================
platform darwin -- Python 3.8.16[pypy-7.3.11-final], pytest-7.0.0, pluggy-1.0.0
C++ Info: Apple LLVM 14.0.0 (clang-1400.0.29.202) C++11 __pybind11_internals_v4_clang_libcpp_cxxabi1002_sh_def__ PYBIND11_SIMPLE_GIL_MANAGEMENT=True
...
=================================== FAILURES ===================================
_ test_run_in_process_multiple_threads_sequential[test_cross_module_gil_inner_pybind11_released] _

test_fn = <function test_cross_module_gil_inner_pybind11_released at 0x00007f967d9694c0>

    @pytest.mark.parametrize("test_fn", ALL_BASIC_TESTS_PLUS_INTENTIONAL_DEADLOCK)
    def test_run_in_process_multiple_threads_sequential(test_fn):
        """Makes sure there is no GIL deadlock when running in a thread multiple times sequentially.
    
        It runs in a separate process to be able to stop and assert if it deadlocks.
        """
>       assert _run_in_process(_run_in_threads, test_fn, num_threads=8, parallel=False) == 0

test_fn    = <function test_cross_module_gil_inner_pybind11_released at 0x00007f967d9694c0>

test_gil_scoped.py:234: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

target = <function _run_in_threads at 0x00007f967d969f60>
args = (<function test_cross_module_gil_inner_pybind11_released at 0x00007f967d9694c0>,)
kwargs = {'num_threads': 8, 'parallel': False}
test_fn = <function test_cross_module_gil_inner_pybind11_released at 0x00007f967d9694c0>
timeout = 10
process = <Process name='Process-45' pid=18326 parent=15388 stopped exitcode=-SIGTERM daemon>
t_start = [167](https://github.com/pybind/pybind11/actions/runs/4121981206/jobs/7119639395#step:12:168)5848907.445822, t_delta = 10.322304964065552, @py_assert1 = None

    def _run_in_process(target, *args, **kwargs):
        if len(args) == 0:
            test_fn = target
        else:
            test_fn = args[0]
        # Do not need to wait much, 10s should be more than enough.
        timeout = 0.1 if test_fn is _intentional_deadlock else 10
        process = multiprocessing.Process(target=target, args=args, kwargs=kwargs)
        process.daemon = True
        try:
            t_start = time.time()
            process.start()
            if timeout >= 100:  # For debugging.
                print(
                    "\nprocess.pid STARTED", process.pid, (sys.argv, target, args, kwargs)
                )
                print(f"COPY-PASTE-THIS: gdb {sys.argv[0]} -p {process.pid}", flush=True)
            process.join(timeout=timeout)
            if timeout >= 100:
                print("\nprocess.pid JOINED", process.pid, flush=True)
            t_delta = time.time() - t_start
            if process.exitcode == 66 and m.defined_THREAD_SANITIZER:  # Issue #2754
                # WOULD-BE-NICE-TO-HAVE: Check that the message below is actually in the output.
                # Maybe this could work:
                # https://gist.github.com/alexeygrigorev/01ce847f2e721b513b42ea4a6c96905e
                pytest.skip(
                    "ThreadSanitizer: starting new threads after multi-threaded fork is not supported."
                )
            elif test_fn is _intentional_deadlock:
                assert process.exitcode is None
                return 0
            elif process.exitcode is None:
                assert t_delta > 0.9 * timeout
                msg = "DEADLOCK, most likely, exactly what this test is meant to detect."
                if env.PYPY and env.WIN:
                    pytest.skip(msg)
>               raise RuntimeError(msg)
E               RuntimeError: DEADLOCK, most likely, exactly what this test is meant to detect.

args       = (<function test_cross_module_gil_inner_pybind11_released at 0x00007f967d9694c0>,)
kwargs     = {'num_threads': 8, 'parallel': False}
msg        = 'DEADLOCK, most likely, exactly what this test is meant to detect.'
process    = <Process name='Process-45' pid=18326 parent=15388 stopped exitcode=-SIGTERM daemon>
t_delta    = 10.322304964065552
t_start    = 1675848907.445822
target     = <function _run_in_threads at 0x00007f967d969f60>
test_fn    = <function test_cross_module_gil_inner_pybind11_released at 0x00007f967d9694c0>
timeout    = 10

test_gil_scoped.py:186: RuntimeError
=========================== short test summary info ============================
SKIPPED [1] test_buffers.py:11: could not import 'numpy': No module named 'numpy'
SKIPPED [1] test_eigen_matrix.py:5: could not import 'numpy': No module named 'numpy'
SKIPPED [1] test_eigen_tensor.py:5: could not import 'numpy': No module named 'numpy'
SKIPPED [1] test_numpy_array.py:6: could not import 'numpy': No module named 'numpy'
SKIPPED [1] test_numpy_dtypes.py:8: could not import 'numpy': No module named 'numpy'
SKIPPED [1] test_numpy_vectorize.py:5: could not import 'numpy': No module named 'numpy'
SKIPPED [1] test_builtin_casters.py:145: no <string_view>
SKIPPED [1] test_builtin_casters.py:320: could not import 'numpy': No module named 'numpy'
SKIPPED [1] test_builtin_casters.py:486: could not import 'numpy': No module named 'numpy'
SKIPPED [1] test_callbacks.py:134: PyPy segfaults on here. See discussion on #1413.
SKIPPED [1] test_callbacks.py:203: Current PYBIND11_INTERNALS_VERSION too low
SKIPPED [1] test_class_sh_basic.py:123: unconditional skip
SKIPPED [1] test_copy_move.py:77: no <optional>
SKIPPED [1] test_custom_type_setup.py:35: condition: env.PYPY
SKIPPED [1] test_custom_type_setup.py:43: condition: env.PYPY
SKIPPED [1] test_exceptions.py:321: PyErr_NormalizeException Segmentation fault
SKIPPED [1] test_modules.py:104: Sufficiently exercised on platforms other than PyPy.
SKIPPED [1] test_pytypes.py:403: Not defined: PYBIND11_HANDLE_REF_DEBUG
SKIPPED [1] test_sequences_and_iterators.py:13: no <optional>
SKIPPED [1] test_stl.py:111: no <optional>
SKIPPED [1] test_stl.py:143: no <experimental/optional>
SKIPPED [1] test_stl.py:234: no <filesystem>
SKIPPED [1] test_stl.py:270: no std::monostate
SKIPPED [1] test_stl_binders.py:89: could not import 'numpy': No module named 'numpy'
SKIPPED [1] test_type_caster_odr_guard_1.py:30: type_caster_odr_guard_registry_values() is None
SKIPPED [1] test_type_caster_odr_guard_1.py:38: type_caster_odr_violation_detected_count() is None
SKIPPED [1] test_type_caster_odr_guard_2.py:9: UNEXPECTED: test_type_caster_odr_guard_2.cpp prevailed (to_python).
XFAIL test_call_policies.py::test_alive_gc
  _PyObject_GetDictPtr is unimplemented
XFAIL test_class.py::test_class_refcount
  condition: env.PYPY
XFAIL test_class.py::test_final
  condition: env.PYPY
XFAIL test_class.py::test_non_final_final
  condition: env.PYPY
XFAIL test_class_sh_disowning_mi.py::test_multiple_inheritance_python
  condition: env.PYPY
XFAIL test_class_sh_disowning_mi.py::test_disown_base1_first[MI2-3-4-1]
  condition: env.PYPY
XFAIL test_class_sh_disowning_mi.py::test_disown_base1_first[MI3-5-6-1]
  condition: env.PYPY
XFAIL test_class_sh_disowning_mi.py::test_disown_base1_first[MI4-7-8-1]
  condition: env.PYPY
XFAIL test_class_sh_disowning_mi.py::test_disown_base2_first[MI2-3-4-1]
  condition: env.PYPY
XFAIL test_class_sh_disowning_mi.py::test_disown_base2_first[MI3-5-6-1]
  condition: env.PYPY
XFAIL test_class_sh_disowning_mi.py::test_disown_base2_first[MI4-7-8-1]
  condition: env.PYPY
XFAIL test_class_sh_disowning_mi.py::test_disown_base2[MI7-13-4]
  condition: env.PYPY
XFAIL test_class_sh_disowning_mi.py::test_disown_base2[MI8b-15-3]
  condition: env.PYPY
XFAIL test_class_sh_property.py::test_valu_getter[m_valu_readonly]
  gc after `del field` is apparently deferred
XFAIL test_class_sh_property.py::test_valu_getter[m_valu_readwrite]
  gc after `del field` is apparently deferred
XFAIL test_eval.py::test_eval_file
  condition: env.PYPY
XFAIL test_exceptions.py::test_cross_module_exception_translator
  See Issue #2847, PR #2999, PR #4324
XFAIL test_exceptions.py::test_python_alreadyset_in_destructor
  Failure on PyPy 3.8 (7.3.7)
XFAIL test_methods_and_attributes.py::test_dynamic_attributes
  condition: env.PYPY
XFAIL test_methods_and_attributes.py::test_cyclic_gc
  condition: env.PYPY
XFAIL test_multiple_inheritance.py::test_multiple_inheritance_mix1
  condition: env.PYPY
XFAIL test_multiple_inheritance.py::test_multiple_inheritance_python
  condition: env.PYPY
XFAIL test_pickling.py::test_roundtrip_with_dict[PickleableWithDict]
  condition: env.PYPY
XFAIL test_pickling.py::test_roundtrip_with_dict[PickleableWithDictNew]
  condition: env.PYPY
XFAIL test_pytypes.py::test_memoryview_refcount[test_memoryview_object]
  getrefcount is not available
XFAIL test_pytypes.py::test_memoryview_refcount[test_memoryview_buffer_info]
  getrefcount is not available
XFAIL test_virtual_functions.py::test_move_support
  condition: env.PYPY
XPASS test_call_policies.py::test_keep_alive_argument sometimes comes out 1 off on PyPy
XPASS test_class_sh_disowning_mi.py::test_disown_base1_first[MI1-1-2-None] condition: env.PYPY
XPASS test_class_sh_disowning_mi.py::test_disown_base1_first[MI5-10-11-1] condition: env.PYPY
XPASS test_class_sh_disowning_mi.py::test_disown_base2_first[MI1-1-2-None] condition: env.PYPY
XPASS test_class_sh_disowning_mi.py::test_disown_base2_first[MI5-10-11-1] condition: env.PYPY
XPASS test_class_sh_disowning_mi.py::test_disown_base2[MI6-12-1] condition: env.PYPY
XPASS test_class_sh_disowning_mi.py::test_disown_base2[MI8-14-1] condition: env.PYPY
FAILED test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_cross_module_gil_inner_pybind11_released]
= 1 failed, 717 passed, 27 skipped, 27 xfailed, 7 xpassed in 182.55s (0:03:02) =

@rwgk
Copy link
Collaborator Author

rwgk commented Feb 22, 2023

PR #4524

https://github.com/pybind/pybind11/actions/runs/4246322249/jobs/7382986308

🐍 3.10 • macos-latest • x64

============================= test session starts ==============================
platform darwin -- Python 3.10.10, pytest-7.0.0, pluggy-1.0.0
C++ Info: Apple LLVM 14.0.0 (clang-1400.0.29.202) C++11 __pybind11_internals_v4_clang_libcpp_cxxabi1002__ PYBIND11_SIMPLE_GIL_MANAGEMENT=True
rootdir: /Users/runner/work/pybind11/pybind11/tests, configfile: pytest.ini
plugins: timeout-2.1.0
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 1002 items

test_async.py ..                                                         [  0%]
test_buffers.py .........                                                [  1%]
test_builtin_casters.py .....s...............                            [  3%]
test_call_policies.py ........                                           [  3%]
test_callbacks.py .............s                                         [  5%]
test_chrono.py ...........................................               [  9%]
test_class.py ................................                           [ 12%]
test_class_sh_basic.py .................................s......          [ 16%]
test_class_sh_disowning.py ...                                           [ 17%]
test_class_sh_disowning_mi.py .........................                  [ 19%]
test_class_sh_factory_constructors.py ......                             [ 20%]
test_class_sh_inheritance.py .......                                     [ 20%]
test_class_sh_mi_thunks.py ..........                                    [ 21%]
test_class_sh_module_local.py ..                                         [ 22%]
test_class_sh_property.py .................                              [ 23%]
test_class_sh_shared_ptr_copy_move.py .....                              [ 24%]
test_class_sh_trampoline_basic.py .....                                  [ 24%]
test_class_sh_trampoline_self_life_support.py .....                      [ 25%]
test_class_sh_trampoline_shared_from_this.py ..............              [ 26%]
test_class_sh_trampoline_shared_ptr_cpp_arg.py ........                  [ 27%]
test_class_sh_trampoline_unique_ptr.py ..                                [ 27%]
test_class_sh_unique_ptr_member.py ...                                   [ 28%]
test_class_sh_virtual_py_cpp_mix.py ...............                      [ 29%]
test_class_sh_void_ptr_capsule.py ........                               [ 30%]
test_classh_mock.py .                                                    [ 30%]
test_const_name.py ......................                                [ 32%]
test_constants_and_functions.py .....                                    [ 33%]
test_copy_move.py ....s...                                               [ 33%]
test_custom_type_casters.py ...                                          [ 34%]
test_custom_type_setup.py ..                                             [ 34%]
test_docstring_options.py .                                              [ 34%]
test_eigen_matrix.py ..............................                      [ 37%]
test_eigen_tensor.py ................................................... [ 42%]
..........................................................               [ 48%]
test_enum.py .........                                                   [ 49%]
test_eval.py ....                                                        [ 49%]
test_exc_namespace_visibility.py .                                       [ 49%]
test_exceptions.py .........................                             [ 52%]
test_factory_constructors.py ...............                             [ 53%]
test_gil_scoped.py ..................................................... [ 59%]
..........F..........................                                    [ 62%]
test_iostream.py ......................                                  [ 64%]
test_kwargs_and_defaults.py ........                                     [ 65%]
test_local_bindings.py ..........                                        [ 66%]
test_methods_and_attributes.py ......................                    [ 68%]
test_modules.py .......                                                  [ 69%]
test_multiple_inheritance.py ..................                          [ 71%]
test_numpy_array.py .................................................... [ 76%]
.....                                                                    [ 77%]
test_numpy_dtypes.py ...............                                     [ 78%]
test_numpy_vectorize.py ........                                         [ 79%]
test_opaque_types.py ...                                                 [ 79%]
test_operator_overloading.py .....                                       [ 80%]
test_pickling.py ........                                                [ 81%]
test_pytypes.py .............................s.......................... [ 86%]
..............................                                           [ 89%]
test_return_value_policy_override.py .......................             [ 91%]
test_sequences_and_iterators.py .s............                           [ 93%]
test_smart_ptr.py .............                                          [ 94%]
test_stl.py ........ss..s.s.........                                     [ 97%]
test_stl_binders.py ..........                                           [ 98%]
test_tagbased_polymorphic.py .                                           [ 98%]
test_thread.py ..                                                        [ 98%]
test_type_caster_odr_guard_1.py ..ss                                     [ 98%]
test_type_caster_odr_guard_2.py s.                                       [ 98%]
test_union.py .                                                          [ 99%]
test_virtual_functions.py ..........                                     [100%]

=================================== FAILURES ===================================
_ test_run_in_process_multiple_threads_sequential[test_cross_module_gil_inner_pybind11_acquired] _

test_fn = <function test_cross_module_gil_inner_pybind11_acquired at 0x10e222320>

    @pytest.mark.parametrize("test_fn", ALL_BASIC_TESTS_PLUS_INTENTIONAL_DEADLOCK)
    def test_run_in_process_multiple_threads_sequential(test_fn):
        """Makes sure there is no GIL deadlock when running in a thread multiple times sequentially.
    
        It runs in a separate process to be able to stop and assert if it deadlocks.
        """
>       assert _run_in_process(_run_in_threads, test_fn, num_threads=8, parallel=False) == 0

test_fn    = <function test_cross_module_gil_inner_pybind11_acquired at 0x10e222320>

test_gil_scoped.py:234: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

target = <function _run_in_threads at 0x10e222950>
args = (<function test_cross_module_gil_inner_pybind11_acquired at 0x10e222320>,)
kwargs = {'num_threads': 8, 'parallel': False}
test_fn = <function test_cross_module_gil_inner_pybind11_acquired at 0x10e222320>
timeout = 10
process = <Process name='Process-46' pid=26332 parent=25287 stopped exitcode=-SIGTERM daemon>
t_start = 1677095556.202225, t_delta = 10.501205921173096, @py_assert1 = None

    def _run_in_process(target, *args, **kwargs):
        if len(args) == 0:
            test_fn = target
        else:
            test_fn = args[0]
        # Do not need to wait much, 10s should be more than enough.
        timeout = 0.1 if test_fn is _intentional_deadlock else 10
        process = multiprocessing.Process(target=target, args=args, kwargs=kwargs)
        process.daemon = True
        try:
            t_start = time.time()
            process.start()
            if timeout >= 100:  # For debugging.
                print(
                    "\nprocess.pid STARTED", process.pid, (sys.argv, target, args, kwargs)
                )
                print(f"COPY-PASTE-THIS: gdb {sys.argv[0]} -p {process.pid}", flush=True)
            process.join(timeout=timeout)
            if timeout >= 100:
                print("\nprocess.pid JOINED", process.pid, flush=True)
            t_delta = time.time() - t_start
            if process.exitcode == 66 and m.defined_THREAD_SANITIZER:  # Issue #2754
                # WOULD-BE-NICE-TO-HAVE: Check that the message below is actually in the output.
                # Maybe this could work:
                # https://gist.github.com/alexeygrigorev/01ce847f2e721b513b42ea4a6c96905e
                pytest.skip(
                    "ThreadSanitizer: starting new threads after multi-threaded fork is not supported."
                )
            elif test_fn is _intentional_deadlock:
                assert process.exitcode is None
                return 0
            elif process.exitcode is None:
                assert t_delta > 0.9 * timeout
                msg = "DEADLOCK, most likely, exactly what this test is meant to detect."
                if env.PYPY and env.WIN:
                    pytest.skip(msg)
>               raise RuntimeError(msg)
E               RuntimeError: DEADLOCK, most likely, exactly what this test is meant to detect.

args       = (<function test_cross_module_gil_inner_pybind11_acquired at 0x10e222320>,)
kwargs     = {'num_threads': 8, 'parallel': False}
msg        = 'DEADLOCK, most likely, exactly what this test is meant to detect.'
process    = <Process name='Process-46' pid=26332 parent=25287 stopped exitcode=-SIGTERM daemon>
t_delta    = 10.501205921173096
t_start    = 1677095556.202225
target     = <function _run_in_threads at 0x10e222950>
test_fn    = <function test_cross_module_gil_inner_pybind11_acquired at 0x10e222320>
timeout    = 10

test_gil_scoped.py:186: RuntimeError
=========================== short test summary info ============================
SKIPPED [1] test_builtin_casters.py:145: no <string_view>
SKIPPED [1] test_callbacks.py:213: Current PYBIND11_INTERNALS_VERSION too low
SKIPPED [1] test_class_sh_basic.py:123: unconditional skip
SKIPPED [1] test_copy_move.py:77: no <optional>
SKIPPED [1] test_pytypes.py:403: Not defined: PYBIND11_HANDLE_REF_DEBUG
SKIPPED [1] test_sequences_and_iterators.py:13: no <optional>
SKIPPED [1] test_stl.py:111: no <optional>
SKIPPED [1] test_stl.py:143: no <experimental/optional>
SKIPPED [1] test_stl.py:234: no <filesystem>
SKIPPED [1] test_stl.py:270: no std::monostate
SKIPPED [1] test_type_caster_odr_guard_1.py:30: type_caster_odr_guard_registry_values() is None
SKIPPED [1] test_type_caster_odr_guard_1.py:38: type_caster_odr_violation_detected_count() is None
SKIPPED [1] test_type_caster_odr_guard_2.py:9: UNEXPECTED: test_type_caster_odr_guard_2.cpp prevailed (to_python).
FAILED test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_cross_module_gil_inner_pybind11_acquired]
============= 1 failed, 988 passed, 13 skipped in 63.10s (0:01:03) =============

@rwgk
Copy link
Collaborator Author

rwgk commented Mar 27, 2023

PR #4570

macos-latest • brew install llvm

============================= test session starts ==============================
platform darwin -- Python 3.11.2, pytest-7.0.0, pluggy-1.0.0
C++ Info: Homebrew Clang 15.0.7 C++17 pybind11_internals_v4_clang_libcpp_cxxabi1002 PYBIND11_SIMPLE_GIL_MANAGEMENT=False

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant