ENH: Python 3.13 support

[lithomas1]

Python 3.13b1 is out, which means its time to start looking into Python 3.13 support!

Steps

• Verify support in dependencies (numpy/Cython)
• Python Dev CI testing (potentially with nogil, JIT?)
• Wheels
• Release and whatsnew
• Regular CI testing

[jwd83]

HI, just curious if there has been any progress getting pandas running on 3.13. I built the JIT enabled version of python 3.13 on windows and went to try it with a number of internal tools but ran into build problems trying to get pandas package installed on 3.13. It seems to fail during the numpy portion of the build. Here is the console output from pip. Apologies if this does not belong here or I have made some other major oversight.

Collecting pandas
  Using cached pandas-2.2.2.tar.gz (4.4 MB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [123 lines of output]
      Collecting meson-python==0.13.1
        Using cached meson_python-0.13.1-py3-none-any.whl.metadata (4.1 kB)
      Collecting meson==1.2.1
        Using cached meson-1.2.1-py3-none-any.whl.metadata (1.7 kB)
      Collecting wheel
        Using cached wheel-0.43.0-py3-none-any.whl.metadata (2.2 kB)
      Collecting Cython==3.0.5
        Using cached Cython-3.0.5-py2.py3-none-any.whl.metadata (3.2 kB)
      Collecting numpy>=2.0.0rc1
        Using cached numpy-2.0.0rc2.tar.gz (18.3 MB)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Installing backend dependencies: started
        Installing backend dependencies: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'error'
        error: subprocess-exited-with-error

        Preparing metadata (pyproject.toml) did not run successfully.
        exit code: 1

        [88 lines of output]
        + C:\Users\jared\OneDrive\Ayrscott\Projects\cpython\PCbuild\amd64\python.exe C:\Users\jared\AppData\Local\Temp\pip-install-__8utyq8\numpy_4569bf49982840bc8c08474080c66a12\vendored-meson\meson\meson.py setup C:\Users\jared\AppData\Local\Temp\pip-install-__8utyq8\numpy_4569bf49982840bc8c08474080c66a12 C:\Users\jared\AppData\Local\Temp\pip-install-__8utyq8\numpy_4569bf49982840bc8c08474080c66a12\.mesonpy-y97ho32v -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=C:\Users\jared\AppData\Local\Temp\pip-install-__8utyq8\numpy_4569bf49982840bc8c08474080c66a12\.mesonpy-y97ho32v\meson-python-native-file.ini
        The Meson build system
        Version: 1.2.99
        Source dir: C:\Users\jared\AppData\Local\Temp\pip-install-__8utyq8\numpy_4569bf49982840bc8c08474080c66a12
        Build dir: C:\Users\jared\AppData\Local\Temp\pip-install-__8utyq8\numpy_4569bf49982840bc8c08474080c66a12\.mesonpy-y97ho32v
        Build type: native build
        Project name: NumPy
        Project version: 2.0.0rc2
        C compiler for the host machine: clang (clang 18.1.6 "clang version 18.1.6")
        C linker for the host machine: clang link 14.40.33811.0
        C++ compiler for the host machine: clang++ (clang 18.1.6 "clang version 18.1.6")
        C++ linker for the host machine: clang++ link 14.40.33811.0
        Cython compiler for the host machine: cython (cython 3.0.10)
        Host machine cpu family: x86_64
        Host machine cpu: x86_64
        Program python found: YES (C:\Users\jared\OneDrive\Ayrscott\Projects\cpython\PCbuild\amd64\python.exe)
        Run-time dependency python found: YES 3.13
        Has header "Python.h" with dependency python: YES
        Compiler for C supports arguments -fno-strict-aliasing: YES
        Compiler for C supports arguments -ftrapping-math: YES
        Test features "SSE SSE2 SSE3" : Supported
        Test features "SSSE3" : Supported
        Test features "SSE41" : Supported
        Test features "POPCNT" : Supported
        Test features "SSE42" : Supported
        Test features "AVX" : Supported
        Test features "F16C" : Supported
        Test features "FMA3" : Supported
        Test features "AVX2" : Supported
        Test features "AVX512F" : Supported
        Test features "AVX512CD" : Supported
        Test features "AVX512_KNL" : Supported
        Test features "AVX512_KNM" : Unsupported due to Arguments "-msse, -msse2, -msse3, -mssse3, -msse4.1, -mpopcnt, -msse4.2, -mavx, -mf16c, -mfma, -mavx2, -mno-mmx, -mavx512f, -mavx512cd, -mavx512er, -mavx512pf, -mavx5124fmaps, -mavx5124vnniw, -mavx512vpopcntdq" are not supported
        Test features "AVX512_SKX" : Supported
        Test features "AVX512_CLX" : Supported
        Test features "AVX512_CNL" : Supported
        Test features "AVX512_ICL" : Supported
        Test features "AVX512_SPR" : Unsupported due to Compiler fails against the test code of "AVX512_SPR"
        Configuring npy_cpu_dispatch_config.h using configuration
        Message:
        CPU Optimization Options
          baseline:
            Requested : min
            Enabled   : SSE SSE2 SSE3
          dispatch:
            Requested : max -xop -fma4
            Enabled   : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL

        Library m found: NO
        Fetching value of define "__MINGW32__" :
        Did not find pkg-config by name 'pkg-config'
        Found Pkg-config: NO
        Run-time dependency scipy-openblas found: NO (tried pkgconfig)
        Run-time dependency mkl found: NO (tried pkgconfig and system)
        Run-time dependency mkl found: NO (tried pkgconfig and system)
        Did not find CMake 'cmake'
        Found CMake: NO
        Run-time dependency openblas found: NO (tried pkgconfig, pkgconfig, pkgconfig, system and cmake)
        Run-time dependency flexiblas found: NO (tried pkgconfig and cmake)
        Run-time dependency blis found: NO (tried pkgconfig and cmake)
        Run-time dependency blas found: NO (tried pkgconfig and system)
        Run-time dependency mkl found: NO (tried pkgconfig and system)
        Run-time dependency openblas found: NO (tried pkgconfig, pkgconfig, pkgconfig, system and cmake)
        Run-time dependency flexiblas found: NO (tried pkgconfig and cmake)
        Run-time dependency lapack found: NO (tried pkgconfig and system)
        Program _build_utils/process_src_template.py found: YES (C:\Users\jared\OneDrive\Ayrscott\Projects\cpython\PCbuild\amd64\python.exe C:\Users\jared\AppData\Local\Temp\pip-install-__8utyq8\numpy_4569bf49982840bc8c08474080c66a12\numpy\_build_utils/process_src_template.py)
        Program _build_utils/tempita.py found: YES (C:\Users\jared\OneDrive\Ayrscott\Projects\cpython\PCbuild\amd64\python.exe C:\Users\jared\AppData\Local\Temp\pip-install-__8utyq8\numpy_4569bf49982840bc8c08474080c66a12\numpy\_build_utils/tempita.py)
        Configuring __config__.py using configuration
        ..\numpy\meson.build:389: WARNING: The variable(s) 'BLAS_INCLUDEDIR', 'BLAS_LIBDIR', 'BLAS_OPENBLAS_CONFIG', 'BLAS_PCFILEDIR', 'BLAS_TYPE_NAME', 'BLAS_VERSION' in the input file 'numpy\__config__.py.in' are not present in the given configuration data.
        Checking for size of "short" : 2
        Checking for size of "int" : 4
        Checking for size of "long" : 4
        Checking for size of "long long" : 8
        Checking for size of "float" : 4
        Checking for size of "double" : 8
        Checking for size of "long double" : 8
        Checking for size of "size_t" : 8
        Checking for size of "size_t" : 8 (cached)
        Checking for size of "wchar_t" : 2
        Checking for size of "off_t" : 4
        Checking for size of "Py_intptr_t" with dependency python: -1
        Checking for size of "PY_LONG_LONG" with dependency python: -1
        Has header "complex.h" : YES
        Checking for type "complex float" : NO

        ..\numpy\_core\meson.build:171:4: ERROR: Problem encountered: "complex.h" header does not include complex type complex float

        A full log can be found at C:\Users\jared\AppData\Local\Temp\pip-install-__8utyq8\numpy_4569bf49982840bc8c08474080c66a12\.mesonpy-y97ho32v\meson-logs\meson-log.txt
        [end of output]

        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: metadata-generation-failed

      Encountered error while generating package metadata.

      See above for output.

      note: This is an issue with the package mentioned above, not pip.
      hint: See above for details.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

[lithomas1]

I can't help you with the compiling numpy part, but I just wanted to point out that you can install numpy nightlies, which should have python 3.13 wheels available.

pip install -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple numpy

[jwd83]

I can't help you with the compiling numpy part, but I just wanted to point out that you can install numpy nightlies, which should have python 3.13 wheels available.

pip install -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple numpy

Hey, thanks for taking the time to reply.

My memory is a little hazy on this now as it was a few weeks back but I believe I tried that approach and was able to get standalone numpy to build using a cherrypicked version but when I went on to and tried to install pandas and requested that it not resolve dependencies (so to use my just built version of numpy) I recall that it seemed to ignore that option and still tried to build whatever specific version of numpy it wanted. It's certainly possible I fudged a CLI option or am misremembering something.

[lysnikolaou]

Hey folks!

I'm planning to spend time on fixing any issues related to the new 3.13 free threaded build of CPython in the upcoming weeks. I've already started testing pandas with it. There's quite a few test failures, but most of them are related to 3.13 and not the free-threaded build itself:

1. There's a few failures related to missing PyQT and PyArrow, since both of those are not yet ready for the free threaded build (I didn't try to install Qt and build PyQT from source, that may be worth trying out?).
2. There's quite a few failures related to the csv module raising a ValueError in a call to register_dialect.
3. The rest of the failures are all related to error messages having changed and thus pytest.raises failing.

(This is all tested with the nightly NumPy and Cython wheels)

I'll start investigating further issues that might be there and let everyone know about my progress. I'd also love to talk to anyone who might be trying this out as well, so that we can collaborate better on this effort.

One remaining question: Should I open a new tracking issue for issues specifically related to free threading or should I use this one?

[lithomas1]

Hi @lysnikolaou,

Thanks for working on this.

I am currently working on fixing regular Python 3.13 test failures in #58776. I'm down to 9 failures there.
I haven't taken a look at free-threading yet, but this is definitely something that's interesting to me.
(thanks for starting on this!)

1. There's a few failures related to missing PyQT and PyArrow, since both of those are not yet ready for the free threaded build (I didn't try to install Qt and build PyQT from source, that may be worth trying out?).

Qt is probably not too important here, but if you're able to investigate

(This is all tested with the nightly NumPy and Cython wheels)

I'll start investigating further issues that might be there and let everyone know about my progress. I'd also love to talk to anyone who might be trying this out as well, so that we can collaborate better on this effort.

One remaining question: Should I open a new tracking issue for issues specifically related to free threading or should I use this one?

Please do, it's probably better to separate the general 3.13 compatibility issues from the free threading stuff

P.S.
I wonder how well pandas does with the Python 3.13 JIT 🤔
Is this something you've taken a look at yet?

[lysnikolaou]

Hey @lithomas1! Happy to collaborate on this!

I am currently working on fixing regular Python 3.13 test failures in #58776. I'm down to 9 failures there.

Great! The rest of the failures appear to be changes in error messages, so should we expect to get that PR in relatively soon?

Qt is probably not too important here, but if you're able to investigate

Agreed. I'll try to dive into pandas-specific issues first and only later spend time on PyQT if it absolutely necessary. It might make sense for us to skip these tests PyQT related tests when PyQT is not installed.

Please do, it's probably better to separate the general 3.13 compatibility issues from the free threading stuff

I'll open a new tracking issue soon.

I wonder how well pandas does with the Python 3.13 JIT 🤔 Is this something you've taken a look at yet?

No, I haven't tried that out.

[lithomas1]

Oops, I forgot to finish my sentence
(meant to say if you're able to take at Arrow stuff, that would be nice, as even though Arrow is optional for now, we are pretty tightly integrated with it).

Other than that, a PR adjusting the error messages would be nice.
I have Python 3.13 CI set up in #58776. If you'd like you could target a PR to my fork, or make a new PR with the Github workflow changes there.

[lysnikolaou]

meant to say if you're able to take at Arrow stuff, that would be nice, as even though Arrow is optional for now, we are pretty tightly integrated with it

I can check that after we're done with the pandas extension modules themselves, sure.

Other than that, a PR adjusting the error messages would be nice.

I'll get started on that then.

[EwoutH]

It would be awesome for users and maintainers of downstream dependencies if wheels could be uploaded to PyPI directly after the first release candidate (currently targeted for Tuesday, 2024-07-30).

[lesteve]

Thanks a lot for your work on this 🙏!

Sorry to be the bearer of bad news, but it seems like the Python 3.13 wheel build has been failing for a few days, I created a separate issue about this #59372 .

[EwoutH]

3.13 rc1 is released! This means the ABI is stable and wheels can be uploaded to PyPI.

Tracking for cibuildwheel: pypa/cibuildwheel#1949

What would be fastest feasible course to get wheels on PyPI? Would a 2.2.3 release with 3.13 wheels be possible?

[mroeschke]

What would be fastest feasible course to get wheels on PyPI? Would a 2.2.3 release with 3.13 wheels be possible?

3.13 support will probably land as part of the pandas 3.0 release (probably Fall 2024)

[EwoutH]

Thanks! If it can't be backported to Pandas 2.x, I would love for some 3.0 pre-releases to include it as well.

cibuildwheel now uses the latest Python 3.13.0rc1 and build Python 3.13 by default (pypa/cibuildwheel#1950). Next cibuildwheel release should be soon.

[EwoutH]

• CI: Update to cibuildwheel 2.20.0 #59401

[EwoutH]

When testing the wheels, about 80 of these types of errors happen, on Python 3.13 only:

  >       assert isinstance(left, ExtensionArray), "left is not an ExtensionArray"
  E       AssertionError: left is not an ExtensionArray

Any idea what causes these? See #59401 for the full logs.

[EwoutH]

It seems similar errors are going on on the main branch (example).

Edit: It seems this was the last passing wheel build, which ran on July 26th. It did include successful Python 3.13 wheel builds.

[EwoutH]

Opened an issue for the error above:

• Python 3.13 wheel builds are failing #59426

[EwoutH]

NumPy 2.1.0rc1 is now released with Python 3.13 support. It could be added as a required dependency when using Python 3.13.

"numpy>=2.1.0rc1; python_version>='3.13'",

[EwoutH]

@mroeschke do you think there's any way to get Python 3.13 wheels on PyPI relatively soon, at least before the stable Python 3.13 release? Since there are many downstream dependencies waiting on Pandas before being able to update workflows, tests and wheels.

[mroeschke]

We have a nightly wheel with 3.13 support here: https://anaconda.org/scientific-python-nightly-wheels/pandas

[mroeschke]

2.2.3 released with wheels supporting 3.13 so I think we can close this issue