[WIP] Python 3.12 [ci skip] #46205

ahesford · 2023-09-23T19:00:53Z

As summer (in the northern hemisphere) turns to autumn, and the leaves begin to turn, so too does the Python version begin to turn... this time to 3.12.

[ci skip]

ahesford · 2023-09-24T11:20:55Z

The list of x86_64, x86_64-musl, aarch64 and i686 failures is now 2:

python3-cypari2
python3-fpylll

ahesford · 2023-09-27T16:21:26Z

I believe it's time to start retiring python3-pyside2 and shiboken2. They were never updated to suport Python 3.11 and now do not function with 3.12; shiboken2 segfaults on many of its tests and also when used during the build of python3-pyside2. Dependants of these packages are:

freecad
pyside2-tools (required by freecad only)
syncplay
cutter (latest version seems to support Qt 6)
hydrus (I have a suspicion the dependencies are done wrong; seems to support PyQt{5,6} and PySide{5,6})
python3-QtPy (checkdepends only; should be able to use PyQt6 and PyQt5 exclusively)

Dependants of python3-QtPy (these should be checked with PyQt6 or PyQt5):

git-cola
python3-jupyter_qtconsole

Related: Upstream explicitly disclaims support for Python 3.12 in PySide 2

EDIT: With an update of shiboken2 and python3-pyside2 to 5.15.10, backporting of some pending Python 3.12 fixes from PySide6, and an ugly CMake hack to allow binding generation for QtQuick widgets, python3-pyside2 builds. This may allow us to continue with freecad and syncplay as is for a bit longer.

ahesford · 2023-09-27T19:05:22Z

I think there is a PyQt5 segfault problem. See, e.g., xbps-src check python3-QtPy.

EDIT: resolved by pulling python3-sip-PyQt5 patches from Fedora.

ahesford · 2023-09-28T18:33:56Z

@leahneukirchen To get Vim to compile, I added --with-python3-stable-abi=3.12. It seems the Python.h header introduces some inline asserts that require linking with the Python library differently than how Vim handles it unless the "limited API" is used. Related issues:

if_python3.c: Fix building dynamic Python3 interpreter with Python 3.12 vim/vim#12660
Cannot include Python.h in 3.12 without linking against the CPython library python/cpython#108502

I don't know if there are any adverse effects here, or whether an earlier version should be specified via the argument.

ahesford · 2023-10-01T23:32:54Z

Most packages aren't run through checks, although I do check some. We don't have the resources to verify that all Python packages pass checks when doing these massive updates.

The new gmpy2 and mpmath pass checks, but I didn't go further than that.

tornaria · 2023-10-02T00:00:16Z

Most packages aren't run through checks, although I do check some. We don't have the resources to verify that all Python packages pass checks when doing these massive updates.

The new gmpy2 and mpmath pass checks, but I didn't go further than that.

Ok, I'll investigate, but in principle I would assume sympy is broken with 3.12. I'd rather void stays in the non-bleeding edge...

I'm working on cypari2 now. Note that fpylll and sagemath should suffer from the same issue: cpython doesn't want us to look inside longint, but they don't want to offer a C alternative (they want us to use PyObject_CallMethod to call some python functions: https://github.com/python/cpython/pull/101270/files).

There seems to be a "cheating" way which is to peek inside the structs, i.e. change x->ob_digit for x->long_value.ob_digit (really, why are they messing up with us?). But they changed this in 3.11 and they changed it again in 3.12 and since they are having fun I'd expect they will change it again in 3.13 just because they want to keep us busy. It seems a better (long term) way would be to use _PyLong_FromByteArray and _PyLong_AsByteArray...

ahesford · 2023-10-02T01:16:11Z

I pushed a couple of upstream changes to support Python 3.12 in sympy, and the only test that still reports a failure is a deprecation warning. (I believe many of the failing tests fixed by one of the patches do nothing more substantial than silence unexpected deprecation warnings, which are meaningless failures anyway.)

I don't have a strong opinion about how to resolve the long representation problem in fpylll, cypari2 or sagemath. Ideally upstream decides on a path forward and we backport it. Going through the Python interpreter seems like a suboptimal approach, but is consistent with CPython's intention to conceal implementation details and allow future optimizations. If our package upstreams don't have an accepted patch, the "cheat" (going through _PyLongValue) seems the least obtrusive, because it most closely preserves the spirit of the original implementation.

tornaria · 2023-10-02T03:19:35Z

I pushed a couple of upstream changes to support Python 3.12 in sympy, and the only test that still reports a failure is a deprecation warning. (I believe many of the failing tests fixed by one of the patches do nothing more substantial than silence unexpected deprecation warnings, which are meaningless failures anyway.)

I don't have a strong opinion about how to resolve the long representation problem in fpylll, cypari2 or sagemath. Ideally upstream decides on a path forward and we backport it. Going through the Python interpreter seems like a suboptimal approach, but is consistent with CPython's intention to conceal implementation details and allow future optimizations. If our package upstreams don't have an accepted patch, the "cheat" (going through _PyLongValue) seems the least obtrusive, because it most closely preserves the spirit of the original implementation.

See: https://github.com/tornaria/void-packages/tree/py312

I'll do fpylll and sagemath tomorrow, and do PRs to upstream that fix. Hopefully I can use the same .h file for all projects, and the api and the 3.12 implementation is taken verbatim from python so it should be easier to move forward in case python 3.13 changes again.

tornaria · 2023-10-02T12:23:26Z

Better link: ahesford/void-packages@py312...tornaria:void-packages:py312

So far cypari and fpylll are working (tested x86_64 and i686). Patches taken from submitted upstream PRs (sagemath/cypari2#138 and fplll/fpylll#254).

@ahesford: if you squash, you may want to rename the commit to "patch and rebuild".

I'm now building sagemath deps. I expect a similar fix will suffice.

tornaria · 2023-10-04T13:02:16Z

f2py in numpy is broken. Steps to reproduce:

With python 3.11, it works:

$ cat test.f
      subroutine foo
          end
$ python3 -m numpy.f2py -c test.f 
/usr/lib/python3.11/site-packages/numpy/f2py/f2py2e.py:686: VisibleDeprecationWarning: distutils has been deprecated since NumPy 1.26.Use the Meson backend instead, or generate wrapperswithout -c and use a custom build script
  builder = build_backend(
running build
running config_cc
INFO: unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
INFO: unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
INFO: build_src
INFO: building extension "untitled" sources
INFO: f2py options: []
INFO: f2py:> /tmp/tmpw74b41uy/src.linux-x86_64-3.11/untitledmodule.c
creating /tmp/tmpw74b41uy/src.linux-x86_64-3.11
Reading fortran codes...
        Reading file 'test.f' (format:fix,strict)
Post-processing...
        Block: untitled
                        Block: foo
Applying post-processing hooks...
  character_backward_compatibility_hook
Post-processing (stage 2)...
Building modules...
    Building module "untitled"...
    Generating possibly empty wrappers"
    Maybe empty "untitled-f2pywrappers.f"
        Constructing wrapper function "foo"...
          foo()
    Wrote C/API module "untitled" to file "/tmp/tmpw74b41uy/src.linux-x86_64-3.11/untitledmodule.c"
INFO:   adding '/tmp/tmpw74b41uy/src.linux-x86_64-3.11/fortranobject.c' to sources.
INFO:   adding '/tmp/tmpw74b41uy/src.linux-x86_64-3.11' to include_dirs.
copying /usr/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.c -> /tmp/tmpw74b41uy/src.linux-x86_64-3.11
copying /usr/lib/python3.11/site-packages/numpy/f2py/src/fortranobject.h -> /tmp/tmpw74b41uy/src.linux-x86_64-3.11
INFO:   adding '/tmp/tmpw74b41uy/src.linux-x86_64-3.11/untitled-f2pywrappers.f' to sources.
INFO: build_src: building npy-pkg config files
/usr/lib/python3.11/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
running build_ext
INFO: customize UnixCCompiler
INFO: customize UnixCCompiler using build_ext
INFO: get_default_fcompiler: matching types: '['arm', 'gnu95', 'intel', 'lahey', 'pg', 'nv', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor', 'fujitsu']'
INFO: customize ArmFlangCompiler
WARN: Could not locate executable armflang
INFO: customize Gnu95FCompiler
INFO: Found executable /usr/bin/gfortran
INFO: customize Gnu95FCompiler
INFO: customize Gnu95FCompiler using build_ext
INFO: building 'untitled' extension
INFO: compiling C sources
INFO: C compiler: x86_64-unknown-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fstack-clash-protection -D_FORTIFY_SOURCE=2 -mtune=generic -O2 -pipe -g -fstack-clash-protection -D_FORTIFY_SOURCE=2 -mtune=generic -O2 -pipe -g -fPIC

creating /tmp/tmpw74b41uy/tmp
creating /tmp/tmpw74b41uy/tmp/tmpw74b41uy
creating /tmp/tmpw74b41uy/tmp/tmpw74b41uy/src.linux-x86_64-3.11
INFO: compile options: '-DNPY_DISABLE_OPTIMIZATION=1 -I/tmp/tmpw74b41uy/src.linux-x86_64-3.11 -I/usr/lib/python3.11/site-packages/numpy/core/include -I/usr/include/python3.11 -c'
INFO: x86_64-unknown-linux-gnu-gcc: /tmp/tmpw74b41uy/src.linux-x86_64-3.11/untitledmodule.c
INFO: x86_64-unknown-linux-gnu-gcc: /tmp/tmpw74b41uy/src.linux-x86_64-3.11/fortranobject.c
INFO: compiling Fortran sources
INFO: Fortran f77 compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /usr/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /usr/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
INFO: compile options: '-I/tmp/tmpw74b41uy/src.linux-x86_64-3.11 -I/usr/lib/python3.11/site-packages/numpy/core/include -I/usr/include/python3.11 -c'
INFO: gfortran:f77: test.f
INFO: gfortran:f77: /tmp/tmpw74b41uy/src.linux-x86_64-3.11/untitled-f2pywrappers.f
f951: Warning: Nonconforming tab character in column 1 of line 2 [-Wtabs]
INFO: /usr/bin/gfortran -Wall -g -Wall -g -shared /tmp/tmpw74b41uy/tmp/tmpw74b41uy/src.linux-x86_64-3.11/untitledmodule.o /tmp/tmpw74b41uy/tmp/tmpw74b41uy/src.linux-x86_64-3.11/fortranobject.o /tmp/tmpw74b41uy/test.o /tmp/tmpw74b41uy/tmp/tmpw74b41uy/src.linux-x86_64-3.11/untitled-f2pywrappers.o -L/usr/lib64/gcc/x86_64-unknown-linux-gnu/12.2.0/../../../../lib64 -L/usr/lib64/gcc/x86_64-unknown-linux-gnu/12.2.0/../../../../lib64 -L/usr/lib64 -lgfortran -o ./untitled.cpython-311-x86_64-linux-gnu.so
Removing build directory /tmp/tmpw74b41uy
$ ls untitled.*.so
untitled.cpython-311-x86_64-linux-gnu.so

With python 3.12, it doesn't work:

$ python -m numpy.f2py -c test.f 
Cannot use distutils backend with Python 3.12, using meson backend instead.Using meson backend
Will pass --lower to f2py
See https://numpy.org/doc/stable/f2py/buildtools/meson.htmlReading fortran codes...
        Reading file 'test.f' (format:fix,strict)
Post-processing...
        Block: untitled
                        Block: foo
Applying post-processing hooks...
  character_backward_compatibility_hook
Post-processing (stage 2)...
Building modules...
    Building module "untitled"...
    Generating possibly empty wrappers"
    Maybe empty "untitled-f2pywrappers.f"
        Constructing wrapper function "foo"...
          foo()
    Wrote C/API module "untitled" to file "./untitledmodule.c"
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/lib/python3.12/site-packages/numpy/f2py/__main__.py", line 5, in <module>
    main()
  File "/usr/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 732, in main
    run_compile()
  File "/usr/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 705, in run_compile
    builder.compile()
  File "/usr/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 130, in compile
    self.write_meson_build(self.build_dir)
  File "/usr/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 108, in write_meson_build
    src = meson_template.generate_meson_build()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 71, in generate_meson_build
    template = Template(self.meson_build_template())
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 44, in meson_build_template
    raise FileNotFoundError(
FileNotFoundError: [Errno 2] Meson build template /usr/lib/python3.12/site-packages/numpy/f2py/_backends/meson.build.template does not exist.

tornaria · 2023-10-04T13:07:52Z

BTW, it's also the case that /usr/bin/f2py is missing, although that happens also with python 3.11. In fact 1.25.2_1 has the binaries but 1.26.0_1 does not.

For sagemath, we only call f2py as a module so 1.26 with python 3.11 is ok.

tornaria · 2023-10-05T03:02:32Z

I have a candidate for sagemath here: ahesford/void-packages@py312...tornaria:void-packages:py312.

I split the patches in two:

36403-python_3.12_part_1.patch long but trivial, taken from Fix AttributeError doctests when output includes a suggestion (part of python 3.12 support) sagemath/sage#36403 (positive review)
python312.patch contains the rest of the commits from sagemath/sage@develop...tornaria:sage:py312

ahesford · 2023-10-06T16:24:38Z

Merged at 2628218. Thanks @tornaria for your diligence in getting sagemath ready.

ychin · 2023-10-11T09:11:34Z

Just to follow up on my earlier comment, you should be able to build Vim with Python 3.12 without need to use tricks like turning on stable ABI now (fixed in vim/vim#13290). The flag to use stable ABI is fine to use btw, it's just unnecessary since you aren't using dynamic library loading anyway.

ahesford · 2023-10-11T12:11:33Z

Thanks for the heads-up.

ahesford marked this pull request as draft September 23, 2023 19:01

ahesford force-pushed the py312 branch 2 times, most recently from 6300546 to 7320569 Compare September 24, 2023 00:57

ahesford force-pushed the py312 branch from 13cadf0 to a112458 Compare September 24, 2023 15:44

ahesford changed the title ~~[WIP] Python 3.12~~ [WIP] Python 3.12 [ci skip] Sep 24, 2023

ahesford force-pushed the py312 branch 13 times, most recently from 36bfe48 to 81f3e70 Compare September 27, 2023 16:16

ahesford force-pushed the py312 branch from 81f3e70 to cadb606 Compare September 27, 2023 19:02

ahesford force-pushed the py312 branch 5 times, most recently from 6e6dd4b to 36da526 Compare September 28, 2023 18:31

ahesford force-pushed the py312 branch 2 times, most recently from 29cc8d9 to 85a0e74 Compare September 28, 2023 19:01

ahesford force-pushed the py312 branch 10 times, most recently from df0ff03 to 73fd7c1 Compare October 4, 2023 01:39

ahesford force-pushed the py312 branch 2 times, most recently from dfa70f7 to b08ed85 Compare October 4, 2023 17:34

ahesford force-pushed the py312 branch from b08ed85 to 1c6cdc8 Compare October 5, 2023 12:30

tornaria mentioned this pull request Oct 5, 2023

python3-gmpy2: patch and rebuild for mpfr-4.2.1 #46192

Closed

ahesford force-pushed the py312 branch 2 times, most recently from 1c6cdc8 to 14cae67 Compare October 6, 2023 14:53

DarkRadiant: rebuild for Python 3.12

c5fdac2

ahesford force-pushed the py312 branch from 14cae67 to c5fdac2 Compare October 6, 2023 16:21

ahesford closed this Oct 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Python 3.12 [ci skip] #46205

[WIP] Python 3.12 [ci skip] #46205

ahesford commented Sep 23, 2023 •

edited

Loading

ahesford commented Sep 24, 2023 •

edited

Loading

ahesford commented Sep 27, 2023 •

edited

Loading

ahesford commented Sep 27, 2023 •

edited

Loading

ahesford commented Sep 28, 2023

ahesford commented Oct 1, 2023

tornaria commented Oct 2, 2023

ahesford commented Oct 2, 2023

tornaria commented Oct 2, 2023

tornaria commented Oct 2, 2023

tornaria commented Oct 4, 2023

tornaria commented Oct 4, 2023

tornaria commented Oct 5, 2023

ahesford commented Oct 6, 2023

ychin commented Oct 11, 2023

ahesford commented Oct 11, 2023

[WIP] Python 3.12 [ci skip] #46205

[WIP] Python 3.12 [ci skip] #46205

Conversation

ahesford commented Sep 23, 2023 • edited Loading

ahesford commented Sep 24, 2023 • edited Loading

ahesford commented Sep 27, 2023 • edited Loading

ahesford commented Sep 27, 2023 • edited Loading

ahesford commented Sep 28, 2023

ahesford commented Oct 1, 2023

tornaria commented Oct 2, 2023

ahesford commented Oct 2, 2023

tornaria commented Oct 2, 2023

tornaria commented Oct 2, 2023

tornaria commented Oct 4, 2023

tornaria commented Oct 4, 2023

tornaria commented Oct 5, 2023

ahesford commented Oct 6, 2023

ychin commented Oct 11, 2023

ahesford commented Oct 11, 2023

ahesford commented Sep 23, 2023 •

edited

Loading

ahesford commented Sep 24, 2023 •

edited

Loading

ahesford commented Sep 27, 2023 •

edited

Loading

ahesford commented Sep 27, 2023 •

edited

Loading