BUILD: Install regression on armv7

[cdce8p]

Installation check

• I have read the installation guide.

Platform

Linux-6.10.4-linuxkit-armv7l-with

Installation Method

pip install

pandas Version

pandas-3.0.0.dev0+1524.g23c497bb2f

Python Version

3.12.5

Installation Logs

For Home Assistant we have an action to build custom musllinux wheels. Building pandas on armv7 started to fail recently with the following error:

  [89/152] Compiling C object pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o
  FAILED: pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o
  cc -Ipandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p -Ipandas/_libs -I../../pandas/_libs -I../../venv-312d/lib/python3.12/site-packages/numpy/core/include -I../../pandas/_libs/include -I/usr/local/include/python3.12 -fvisibility=hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -std=c11 -O3 -DNPY_NO_DEPRECATED_API=0 -DNPY_TARGET_VERSION=NPY_1_21_API_VERSION -fPIC -MD -MQ pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o -MF pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o.d -o pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o -c ../../pandas/_libs/src/vendored/ujson/python/JSONtoObj.c
  In file included from /usr/include/fortify/string.h:26,
                   from ../../pandas/_libs/include/pandas/portable.h:12,
                   from ../../pandas/_libs/include/pandas/vendored/ujson/lib/ultrajson.h:55,
                   from ../../pandas/_libs/src/vendored/ujson/python/JSONtoObj.c:41:
  /usr/include/fortify/stdlib.h:42:1: error: 'realpath' undeclared here (not in a function)
     42 | _FORTIFY_FN(realpath) char *realpath(const char *__p, char *__r)
        | ^~~~~~~~~~~
  In file included from /usr/local/include/python3.12/Python.h:23,
                   from ../../pandas/_libs/src/vendored/ujson/python/JSONtoObj.c:43:
  /usr/include/fortify/stdlib.h: In function 'realpath':
  /usr/include/fortify/stdlib.h:45:2: error: #error PATH_MAX unset. A fortified realpath will not work.
     45 | #error PATH_MAX unset. A fortified realpath will not work.
        |  ^~~~~

I bisected the issue to #55150 (cc @WillAyd), in particular the change in pandas/_libs/src/vendored/ujson/python/JSONtoObj.c:

...
 #define PY_ARRAY_UNIQUE_SYMBOL UJSON_NUMPY
 #define NO_IMPORT_ARRAY
 #define PY_SSIZE_T_CLEAN
+#include "pandas/vendored/ujson/lib/ultrajson.h"
 #include <Python.h>
 #include <numpy/arrayobject.h>
-#include "pandas/vendored/ujson/lib/ultrajson.h"
...

It seems that, at least on the particular platform, the include order is important.

For the current main branch, it seems to be enough to modify this part:

pandas/pandas/_libs/src/vendored/ujson/python/JSONtoObj.c

Lines 41 to 43 in 23c497b

	#include "pandas/vendored/ujson/lib/ultrajson.h"
	#define PY_SSIZE_T_CLEAN
	#include <Python.h>

-#include "pandas/vendored/ujson/lib/ultrajson.h"
 #define PY_SSIZE_T_CLEAN
+// clang-format off
 #include <Python.h>
+#include "pandas/vendored/ujson/lib/ultrajson.h"
+// clang-format on

/usr/src# pip install -v .
Using pip 24.2 from /usr/src/venv-312d/lib/python3.12/site-packages/pip (python 3.12)
Processing /usr/src
  Running command Preparing metadata (pyproject.toml)
  + meson setup /usr/src /usr/src/.mesonpy-09kmtf13/build -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --vsenv --native-file=/usr/src/.mesonpy-09kmtf13/build/meson-python-native-file.ini
  The Meson build system
  Version: 1.2.1
  Source dir: /usr/src
  Build dir: /usr/src/.mesonpy-09kmtf13/build
  Build type: native build
  Project name: pandas
  Project version: 3.0.0.dev0+1524.g23c497bb2f
  C compiler for the host machine: cc (gcc 13.2.1 "cc (Alpine 13.2.1_git20240309) 13.2.1 20240309")
  C linker for the host machine: cc ld.bfd 2.42
  C++ compiler for the host machine: c++ (gcc 13.2.1 "c++ (Alpine 13.2.1_git20240309) 13.2.1 20240309")
  C++ linker for the host machine: c++ ld.bfd 2.42
  Cython compiler for the host machine: cython (cython 3.0.11)
  Host machine cpu family: arm
  Host machine cpu: armv7l
  Program python found: YES (/usr/src/venv-312d/bin/python)
  Found pkg-config: /usr/bin/pkg-config (2.2.0)
  Run-time dependency python found: YES 3.12
  Build targets in project: 54

  pandas 3.0.0.dev0+1524.g23c497bb2f

    User defined options
      Native files: /usr/src/.mesonpy-09kmtf13/build/meson-python-native-file.ini
      buildtype   : release
      vsenv       : True
      b_ndebug    : if-release
      b_vscrt     : md

  Found ninja-1.11.1.git.kitware.jobserver-1 at /usr/src/venv-312d/bin/ninja

  Visual Studio environment is needed to run Ninja. It is recommended to use Meson wrapper:
  /usr/src/venv-312d/bin/meson compile -C .
  + /usr/src/venv-312d/bin/ninja
  [1/152] Generating pandas/_libs/index_class_helper_pxi with a custom command
  [2/152] Generating pandas/_libs/khash_primitive_helper_pxi with a custom command
  [3/152] Generating pandas/_libs/algos_common_helper_pxi with a custom command
  [4/152] Generating pandas/_libs/hashtable_func_helper_pxi with a custom command
  [5/152] Generating pandas/_libs/intervaltree_helper_pxi with a custom command
  [6/152] Generating pandas/_libs/hashtable_class_helper_pxi with a custom command
  [7/152] Generating pandas/_libs/algos_take_helper_pxi with a custom command
  [8/152] Generating pandas/__init__.py with a custom command
  [9/152] Generating pandas/_libs/sparse_op_helper_pxi with a custom command
  [10/152] Generating write_version_file with a custom command
  [11/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/base.pyx
  [12/152] Compiling C object pandas/_libs/tslibs/parsing.cpython-312-arm-linux-musleabihf.so.p/.._src_parser_tokenizer.c.o
  [13/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/ccalendar.pyx
  [14/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/dtypes.pyx
  [15/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/nattype.pyx
  [16/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/np_datetime.pyx
  [17/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/fields.pyx
  [18/152] Compiling C object pandas/_libs/tslibs/base.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_base.pyx.c.o
  [19/152] Linking target pandas/_libs/tslibs/base.cpython-312-arm-linux-musleabihf.so
  [20/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/parsing.pyx
  [21/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/conversion.pyx
  [22/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/strptime.pyx
  [23/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/timezones.pyx
  [24/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/timedeltas.pyx
  [25/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/offsets.pyx
  [26/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/period.pyx
  [27/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/timestamps.pyx
  [28/152] Compiling C object pandas/_libs/tslibs/ccalendar.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_ccalendar.pyx.c.o
  [29/152] Linking target pandas/_libs/tslibs/ccalendar.cpython-312-arm-linux-musleabihf.so
  [30/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/tzconversion.pyx
  [31/152] Compiling C object pandas/_libs/tslibs/np_datetime.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_np_datetime.pyx.c.o
  [32/152] Linking target pandas/_libs/tslibs/np_datetime.cpython-312-arm-linux-musleabihf.so
  [33/152] Compiling Cython source /usr/src/pandas/_libs/tslibs/vectorized.pyx
  [34/152] Compiling C object pandas/_libs/tslibs/dtypes.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_dtypes.pyx.c.o
  [35/152] Linking target pandas/_libs/tslibs/dtypes.cpython-312-arm-linux-musleabihf.so
  [36/152] Compiling C object pandas/_libs/tslibs/nattype.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_nattype.pyx.c.o
  [37/152] Linking target pandas/_libs/tslibs/nattype.cpython-312-arm-linux-musleabihf.so
  [38/152] Compiling Cython source /usr/src/pandas/_libs/arrays.pyx
  [39/152] Compiling C object pandas/_libs/tslibs/conversion.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_conversion.pyx.c.o
  [40/152] Linking target pandas/_libs/tslibs/conversion.cpython-312-arm-linux-musleabihf.so
  [41/152] Compiling C object pandas/_libs/tslibs/timezones.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_timezones.pyx.c.o
  [42/152] Linking target pandas/_libs/tslibs/timezones.cpython-312-arm-linux-musleabihf.so
  [43/152] Compiling C object pandas/_libs/tslibs/fields.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_fields.pyx.c.o
  [44/152] Linking target pandas/_libs/tslibs/fields.cpython-312-arm-linux-musleabihf.so
  [45/152] Compiling C object pandas/_libs/tslibs/vectorized.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_vectorized.pyx.c.o
  [46/152] Linking target pandas/_libs/tslibs/vectorized.cpython-312-arm-linux-musleabihf.so
  [47/152] Compiling Cython source /usr/src/pandas/_libs/algos.pyx
  [48/152] Compiling C object pandas/_libs/tslibs/tzconversion.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_tzconversion.pyx.c.o
  [49/152] Compiling Cython source /usr/src/pandas/_libs/hashing.pyx
  [50/152] Linking target pandas/_libs/tslibs/tzconversion.cpython-312-arm-linux-musleabihf.so
  [51/152] Compiling C object pandas/_libs/arrays.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_arrays.pyx.c.o
  [52/152] Linking target pandas/_libs/arrays.cpython-312-arm-linux-musleabihf.so
  [53/152] Compiling C object pandas/_libs/tslibs/strptime.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_strptime.pyx.c.o
  [54/152] Compiling Cython source /usr/src/pandas/_libs/indexing.pyx
  [55/152] Linking target pandas/_libs/tslibs/strptime.cpython-312-arm-linux-musleabihf.so
  [56/152] Compiling C object pandas/_libs/tslibs/parsing.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_parsing.pyx.c.o
  [57/152] Linking target pandas/_libs/tslibs/parsing.cpython-312-arm-linux-musleabihf.so
  [58/152] Compiling C object pandas/_libs/indexing.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_indexing.pyx.c.o
  [59/152] Linking target pandas/_libs/indexing.cpython-312-arm-linux-musleabihf.so
  [60/152] Compiling Cython source /usr/src/pandas/_libs/groupby.pyx
  [61/152] Compiling Cython source /usr/src/pandas/_libs/internals.pyx
  [62/152] Compiling Cython source /usr/src/pandas/_libs/index.pyx
  [63/152] Compiling C object pandas/_libs/lib.cpython-312-arm-linux-musleabihf.so.p/src_parser_tokenizer.c.o
  [64/152] Compiling C object pandas/_libs/tslibs/period.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_period.pyx.c.o
  [65/152] Linking target pandas/_libs/tslibs/period.cpython-312-arm-linux-musleabihf.so
  [66/152] Compiling Cython source /usr/src/pandas/_libs/hashtable.pyx
  [67/152] Compiling Cython source /usr/src/pandas/_libs/missing.pyx
  [68/152] Compiling C object pandas/_libs/pandas_datetime.cpython-312-arm-linux-musleabihf.so.p/src_vendored_numpy_datetime_np_datetime.c.o
  [69/152] Compiling Cython source /usr/src/pandas/_libs/lib.pyx
  [70/152] Compiling Cython source /usr/src/pandas/_libs/join.pyx
  [71/152] Compiling C object pandas/_libs/hashing.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_hashing.pyx.c.o
  [72/152] Linking target pandas/_libs/hashing.cpython-312-arm-linux-musleabihf.so
  [73/152] Compiling C object pandas/_libs/pandas_datetime.cpython-312-arm-linux-musleabihf.so.p/src_vendored_numpy_datetime_np_datetime_strings.c.o
  [74/152] Compiling C object pandas/_libs/pandas_datetime.cpython-312-arm-linux-musleabihf.so.p/src_datetime_date_conversions.c.o
  [75/152] Compiling Cython source /usr/src/pandas/_libs/interval.pyx
  [76/152] Compiling C object pandas/_libs/pandas_datetime.cpython-312-arm-linux-musleabihf.so.p/src_datetime_pd_datetime.c.o
  [77/152] Linking target pandas/_libs/pandas_datetime.cpython-312-arm-linux-musleabihf.so
  [78/152] Compiling C object pandas/_libs/pandas_parser.cpython-312-arm-linux-musleabihf.so.p/src_parser_io.c.o
  [79/152] Compiling C object pandas/_libs/pandas_parser.cpython-312-arm-linux-musleabihf.so.p/src_parser_pd_parser.c.o
  [80/152] Compiling C object pandas/_libs/pandas_parser.cpython-312-arm-linux-musleabihf.so.p/src_parser_tokenizer.c.o
  [81/152] Linking target pandas/_libs/pandas_parser.cpython-312-arm-linux-musleabihf.so
  [82/152] Compiling C object pandas/_libs/parsers.cpython-312-arm-linux-musleabihf.so.p/src_parser_tokenizer.c.o
  [83/152] Compiling C object pandas/_libs/parsers.cpython-312-arm-linux-musleabihf.so.p/src_parser_io.c.o
  [84/152] Compiling C object pandas/_libs/tslibs/timedeltas.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_timedeltas.pyx.c.o
  [85/152] Linking target pandas/_libs/tslibs/timedeltas.cpython-312-arm-linux-musleabihf.so
  [86/152] Compiling C object pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_ujson.c.o
  [87/152] Compiling C object pandas/_libs/tslibs/timestamps.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_timestamps.pyx.c.o
  [88/152] Linking target pandas/_libs/tslibs/timestamps.cpython-312-arm-linux-musleabihf.so
  [89/152] Compiling C object pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o
  FAILED: pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o
  cc -Ipandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p -Ipandas/_libs -I../../pandas/_libs -I../../venv-312d/lib/python3.12/site-packages/numpy/core/include -I../../pandas/_libs/include -I/usr/local/include/python3.12 -fvisibility=hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -std=c11 -O3 -DNPY_NO_DEPRECATED_API=0 -DNPY_TARGET_VERSION=NPY_1_21_API_VERSION -fPIC -MD -MQ pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o -MF pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o.d -o pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_JSONtoObj.c.o -c ../../pandas/_libs/src/vendored/ujson/python/JSONtoObj.c
  In file included from /usr/include/fortify/string.h:26,
                   from ../../pandas/_libs/include/pandas/portable.h:12,
                   from ../../pandas/_libs/include/pandas/vendored/ujson/lib/ultrajson.h:55,
                   from ../../pandas/_libs/src/vendored/ujson/python/JSONtoObj.c:41:
  /usr/include/fortify/stdlib.h:42:1: error: 'realpath' undeclared here (not in a function)
     42 | _FORTIFY_FN(realpath) char *realpath(const char *__p, char *__r)
        | ^~~~~~~~~~~
  In file included from /usr/local/include/python3.12/Python.h:23,
                   from ../../pandas/_libs/src/vendored/ujson/python/JSONtoObj.c:43:
  /usr/include/fortify/stdlib.h: In function 'realpath':
  /usr/include/fortify/stdlib.h:45:2: error: #error PATH_MAX unset. A fortified realpath will not work.
     45 | #error PATH_MAX unset. A fortified realpath will not work.
        |  ^~~~~
  [90/152] Compiling C object pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_lib_ultrajsonenc.c.o
  [91/152] Compiling C object pandas/_libs/json.cpython-312-arm-linux-musleabihf.so.p/src_vendored_ujson_python_objToJSON.c.o
  [92/152] Compiling Cython source /usr/src/pandas/_libs/parsers.pyx
  [93/152] Compiling C object pandas/_libs/missing.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_missing.pyx.c.o
  [94/152] Compiling C object pandas/_libs/internals.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_internals.pyx.c.o
  [95/152] Compiling C object pandas/_libs/tslibs/offsets.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_tslibs_offsets.pyx.c.o
  [96/152] Compiling C object pandas/_libs/index.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_index.pyx.c.o
  [97/152] Compiling C object pandas/_libs/lib.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_lib.pyx.c.o
  [98/152] Compiling C object pandas/_libs/join.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_join.pyx.c.o
  [99/152] Compiling C object pandas/_libs/interval.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_interval.pyx.c.o
  [100/152] Compiling C object pandas/_libs/algos.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_algos.pyx.c.o
  [101/152] Compiling C object pandas/_libs/groupby.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_groupby.pyx.c.o
  [102/152] Compiling C object pandas/_libs/hashtable.cpython-312-arm-linux-musleabihf.so.p/meson-generated_pandas__libs_hashtable.pyx.c.o
  ninja: build stopped: subcommand failed.
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /usr/src/venv-312d/bin/python /usr/src/venv-312d/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp827zleqr
  cwd: /usr/src
  Preparing metadata (pyproject.toml) ... error
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.