Shipped Boost.Python is incompatible with Python 3.11

[easybob95]

Hello,

in order to test Python 3.11 with pycuda, i have just installed latest pycuda version : 2022.1

I use CUDA 11.6.2 with a windows 11 laptop.

When i try my program (using pycuda), i get this error message :

Traceback (most recent call last):
  File "D:\Alain\Astro\Soft\PC\PC_Traitement_Video_V7_51g.py", line 19, in <module>
    import pycuda.driver as drv
  File "C:\Users\apail\AppData\Local\Programs\Python\Python311\Lib\site-packages\pycuda\driver.py", line 65, in <module>
    from pycuda._driver import *  # noqa
SystemError: type Boost.Python.enum has the Py_TPFLAGS_HAVE_GC flag but has no traverse function

Any help will be appreciate.

Alain

[inducer]

Thanks for the report! My read on this is that the (quite old, quite long in the tooth version of Boost.Python that ships with PyCUDA is incompatible with Python 3.11. You should be able to switch to Boost.Python from an up-to-date version of Boost for an immediate remedy.

In the medium-to-short term, we should either update the shipped Boost.Python or migrate to Pybind11 or nanobind. I'm leaning towards one of the latter.

[easybob95]

Hello,

i installed boost python 1.78 but i get the same issue.

Alain

[inducer]

I saw you filed boostorg/python#388.

If you're sure you disabled the shipped boost and the updated Boost.Python was used instead, then it may just be that Boost.Python itself still needs to be updated.

[easybob95]

Hello,
i made some tests and i think i still have the issue even with the updated Boost.Python.

Alain

[easybob95]

Hello,

any good or bad news about this problem ?

Alain

[inducer]

We could try simply applying this patch if that works: boostorg/python@a218bab.

[easybob95]

This means you will release a new pycuda version for Python 3.11 ?

If so, i will test it.

Alain

[inducer]

No, that means I'm hoping you can apply that patch to the source under bpl-subset and report back.

[easybob95]

I am really honoured you think i can do such thing. But to be honest, i don't understand a single word about the solution to this issue.
I am very sorry to say that i will never succed applying that patch and i do regret that. Really.

Alain

[inducer]

Fair enough. I've included the patch in 697139f. It would be helpful if you could try and build that.

[inducer]

(and report back)

[easybob95]

Hello Andreas,

Many thanks.

As i am one of the worst software writer, i must say you still use Chinese language (for example, my best level is to type "py -m pip install pycuda-XXX-XXX-XXX.whl").

So i will study the build of a modified boost (i guess ?).

For now, my holidays are coming and my software works fine on my laptop. As i plan to make many deep sky observations and videos using my software (it's ok with Python 3.10 and Pycuda 2022.1), i won't touch anything. If i make some experiments, as i don't really understand what i am supposed to do, i think i am able to destroy everything (and i would like to avoid this).

I am sorry Andreas to be such noob. This does not help me when i try to solve some issues with my system and i am my main and first victim !

Anyway, i really appreciate your help.

I'll be back.

Alain

[inducer]

Make sure that PyCUDA is not currently installed, then try

pip install git+https://github.com/inducer/pycuda.git

[easybob95]

I made a test.

So, i first tried this with Python3.10. I was able to install pycuda and everything was fine.

I removed everything and i tried with Python 3.11 RC1.

Compilation failed.

I get many warnings and finally errors :

The last lines of the compilation :

  "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DBOOST_ALL_NO_LIB=1 -DBOOST_THREAD_BUILD_DLL=1 -DBOOST_MULTI_INDEX_DISABLE_SERIALIZATION=1 -DBOOST_PYTHON_SOURCE=1 -Dboost=pycudaboost -DBOOST_THREAD_DONT_USE_CHRONO=1 -DPYGPU_PACKAGE=pycuda -DPYGPU_PYCUDA=1 -DHAVE_CURAND=1 -Isrc/cpp -Ibpl-subset/bpl_subset "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\include" -ID:\Alain\TEMP\pip-build-env-pjrbi85s\overlay\Lib\site-packages\numpy\core\include -IC:\Users\apail\AppData\Local\Programs\Python\Python311\include -IC:\Users\apail\AppData\Local\Programs\Python\Python311\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /EHsc /Tpbpl-subset/bpl_subset/libs/python/src/numeric.cpp /Fobuild\temp.win-amd64-cpython-311\Release\bpl-subset/bpl_subset/libs/python/src/numeric.obj /EHsc
  numeric.cpp
  Unknown compiler version - please run the configure tests and report the results
  bpl-subset/bpl_subset\boost/python/detail/caller.hpp(55): warning C4244: 'return'ÿ: conversion de 'Py_ssize_t' en 'unsigned int', perte possible de donn‚es
  bpl-subset/bpl_subset\pycudaboost/python/call.hpp(62): warning C4996: 'PyEval_CallFunction': deprecated in 3.9
  bpl-subset/bpl_subset\pycudaboost/python/object_call.hpp(19): note: voir la r‚f‚rence … l'instanciation de la fonction modŠle 'pycudaboost::python::api::object pycudaboost::python::call<obj,A0>(PyObject *,const A0 &,pycudaboost::type<obj> *)' en cours de compilation
          with
          [
              A0=pycudaboost::python::api::object
          ]
  bpl-subset/bpl_subset\pycudaboost/preprocessor/iteration/detail/local.hpp(37): note: voir la r‚f‚rence … l'instanciation de la fonction modŠle 'pycudaboost::python::api::object pycudaboost::python::api::object_operators<pycudaboost::python::api::object>::operator ()<pycudaboost::python::api::object>(const A0 &) const' en cours de compilation
          with
          [
              A0=pycudaboost::python::api::object
          ]
  bpl-subset/bpl_subset\pycudaboost/preprocessor/iteration/detail/local.hpp(37): note: voir la r‚f‚rence … l'instanciation de la fonction modŠle 'pycudaboost::python::api::object pycudaboost::python::api::object_operators<pycudaboost::python::api::object>::operator ()<pycudaboost::python::api::object>(const A0 &) const' en cours de compilation
          with
          [
              A0=pycudaboost::python::api::object
          ]
  "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DBOOST_ALL_NO_LIB=1 -DBOOST_THREAD_BUILD_DLL=1 -DBOOST_MULTI_INDEX_DISABLE_SERIALIZATION=1 -DBOOST_PYTHON_SOURCE=1 -Dboost=pycudaboost -DBOOST_THREAD_DONT_USE_CHRONO=1 -DPYGPU_PACKAGE=pycuda -DPYGPU_PYCUDA=1 -DHAVE_CURAND=1 -Isrc/cpp -Ibpl-subset/bpl_subset "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\include" -ID:\Alain\TEMP\pip-build-env-pjrbi85s\overlay\Lib\site-packages\numpy\core\include -IC:\Users\apail\AppData\Local\Programs\Python\Python311\include -IC:\Users\apail\AppData\Local\Programs\Python\Python311\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30037\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /EHsc /Tpbpl-subset/bpl_subset/libs/python/src/object/class.cpp /Fobuild\temp.win-amd64-cpython-311\Release\bpl-subset/bpl_subset/libs/python/src/object/class.obj /EHsc
  class.cpp
  Unknown compiler version - please run the configure tests and report the results
  bpl-subset/bpl_subset\boost/python/detail/caller.hpp(55): warning C4244: 'return'ÿ: conversion de 'Py_ssize_t' en 'unsigned int', perte possible de donn‚es
  bpl-subset/bpl_subset/libs/python/src/object/class.cpp(211): error C2106: '='ÿ: l'op‚rande gauche doit ˆtre une l-value
  bpl-subset/bpl_subset/libs/python/src/object/class.cpp(319): error C2106: '='ÿ: l'op‚rande gauche doit ˆtre une l-value
  bpl-subset/bpl_subset/libs/python/src/object/class.cpp(382): error C2106: '='ÿ: l'op‚rande gauche doit ˆtre une l-value
  bpl-subset/bpl_subset/libs/python/src/object/class.cpp(473): error C2106: '='ÿ: l'op‚rande gauche doit ˆtre une l-value
  bpl-subset/bpl_subset/libs/python/src/object/class.cpp(734): warning C4267: 'initialisation'ÿ: conversion de 'size_t' en 'int', perte possible de donn‚es
  bpl-subset/bpl_subset/libs/python/src/object/class.cpp(742): error C2106: '='ÿ: l'op‚rande gauche doit ˆtre une l-value
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.29.30037\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pycuda
Failed to build pycuda
ERROR: Could not build wheels for pycuda, which is required to install pyproject.toml-based projects

[inducer]

Hmm, OK. So there might not be an easy win here.

[easybob95]

Nevermind, things are ok with Python 3.10.

I have a working system to make some interesting videos of the deep sky. That's because of your great work !

Alain

[AlainPaillou]

Hello,

any news (good or bad) about this issue ?

Alain

[jakirkham]

Sorry trying to get caught up here and am likely still missing context, it sounds like the vendored Boost has been updated and maybe works? If that's true, does this come down to making a new release? Or if I'm misunderstanding things (quite likely), what is still needed here (and are there things others can do to help)?

[anbe42]

The Debian pycuda package started failing to build when python 3.11 was added as a supported version a few days ago.
We use the system boost (which should be 1.74) and not bpl-subset.

https://salsa.debian.org/nvidia-team/python-pycuda/-/jobs/3525934#L4023

In file included from /usr/include/python3.11/Python.h:44:
src/wrapper/_pvt_struct_v3.cpp: In function ‘PyObject* PyInit__pvt_struct()’:
/usr/include/python3.11/object.h:136:30: error: lvalue required as left operand of assignment
  136 | #  define Py_TYPE(ob) Py_TYPE(_PyObject_CAST(ob))
      |                       ~~~~~~~^~~~~~~~~~~~~~~~~~~~
src/wrapper/_pvt_struct_v3.cpp:1733:5: note: in expansion of macro ‘Py_TYPE’
 1733 |     Py_TYPE(&PyStructType) = &PyType_Type;
      |     ^~~~~~~
error: command '/usr/lib/ccache/x86_64-linux-gnu-gcc' failed with exit code 1
E: pybuild pybuild:379: build: plugin distutils failed with: exit code=1: /usr/bin/python3.11 setup.py build 

I see the same when building from the pycuda main branch.

[inducer]

This latter failure is independent of Boost. It should be addressed by #395. With that PR merged, PyCUDA main should be buildable against up-to-date Boost (though not yet the shipped version).

[inducer]

#397 should put this issue to bed, I think. The migration to pybind (#393) should still happen IMO, but this buys us time to do that.

[inducer]

https://pypi.org/project/pycuda/2022.2/

[easybob95]

pycuda 2022.2 still does not work for me with Python 3.11. I get an error when loading pycuda with Python.

Anyway, i give up and stay with Python 3.10

Alain

[inducer]

What's the error message?

[easybob95]

Nothing special or clear. Python just says there is an error here :
from pycuda._driver import *

It can't import pycuda.driver

A bit strange. It seems am the only one who have an issue with Python 3.11

As long as Python 3.10 is ok, it is not a big problem for me.

Have a nice day.

Alain

[jakirkham]

Is it possible that there's some cached files that are lingering? Maybe it is worth purging everything and starting again fresh?

[easybob95]

I will uninstall everything and make a new clean install to see if things are better.

Alain

[easybob95]

Hello,

i removed everything for Python 3.10 and Python 3.11

I made a clean install with Python 3.11 and al the libraries i need.

I installed Pycuda 2022.2 without error.

When i run my program with Python 3.11, here is the result :

========= RESTART: D:\Alain\Astro\Soft\PC\Treatment_V17_01d_VPI_Cupy.py ========
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf
Traceback (most recent call last):
File "D:\Alain\Astro\Soft\PC\Treatment_V17_01d_VPI_Cupy.py", line 36, in
import pycuda.driver as drv
File "C:\Users\apail\AppData\Local\Programs\Python\Python311\Lib\site-packages\pycuda\driver.py", line 66, in
from pycuda._driver import * # noqa
SystemError: initialization of _driver raised unreported exception

This makes me crazy. Really.

Alain

[inducer]

My suspicion in your case would be numpy headers being found that don't match the installed numpy binary.

[jakirkham]

So pip/conda install NumPy first and then install PyCUDA?

[easybob95]

Well guys, good news : it works (at least).

It seems i had some problems with libraries i get there : https://www.lfd.uci.edu/~gohlke/pythonlibs/

So, i removed everything one more time and i directly get the libs from the web using pip.

Many many thanks guys for your help !

Now, i will try to not bother you any more !

You rock guys. Thx !

Alain