[WIP] dissociate python optional test wheels builds + python313 #6282

th0ma7 · 2024-10-14T20:01:25Z

Description

Intent is to:

Remove wheel build testing from python310/python311 base Makefiles to ease updates
Add python313 package (potentially also add python312)
Dissociate crossenv creation from python3*/Makefile -> Now uses spksrc.crossenv.mk

$ ls -la ~/spksrc/mk/crossenv/requirements-default.txt
$ make crossenvclean
$ make crossenc-<arch>-<version>

Allow wheel specific crossenv enablement using a mk/crossenv/ directory containing wheel crossenv definitions:

$ ls -la ~/spksrc/mk/crossenv/requirements-aiohttp-3.8.5.txt
$ make crossenvclean
$ WHEEL="aiohttp-3.8.5" make crossenv-x64-7.1

Checklist

Build rule all-supported completed successfully
New installation of package completed successfully
Package upgrade completed successfully (Manually install the package again)
Package functionality was tested
Any needed documentation is updated/created

Type of change

TODO

Migrate spksrc.wheel.mk to use $(which cross-pip)
Remove / fix / validate usage of pip cache through $HOME/.cache/pip

2024-11-14T02:37:23.1044954Z WARNING: The directory '/github/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.

Fix handling or requirements.txt entry with trailing comment test==1234 # This is a test wheel
Have pure requirements wheels to use caching similarly to crossenv wheels
Review OPENSSL_*_DIR variables and logic usage throughout python related mk files
Use status cookie for crossenv creation
Use status cookie for wheel building

cross/python313/Makefile

th0ma7 · 2024-10-14T23:34:58Z

@hgy59 I'll let this run but this patch in particular 9c764a7 needs to be in its own PR... interesting from not having noticed it earlier. I'll also mark as [WIP] as not even close to be merged, this was early work being push to initiate exchange with python github relatively to the build issue I had.

th0ma7 · 2024-10-15T11:18:43Z

@hgy59 bind compiling tries to install packages ?!?!? I'll try to find where that comes from but in case it ring any bell...

===>  Patching for bind
===>  Configuring for bind
===>  - Configure ARGS: --host=x86_64-pc-linux-gnu --build=i686-pc-linux --prefix=/usr/local/bind --localstatedir=/usr/local/bind/var --enable-full-report BUILD_CC=cc BUILD_CFLAGS=-I/home/spksrc/python-wheels/spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/usr/include -I/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/include -O2 -O2 BUILD_CPPFLAGS=-I/home/spksrc/python-wheels/spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/usr/include -I/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/include -O2 -O2 BUILD_LDFLAGS=-L/home/spksrc/python-wheels/spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/lib -L/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/lib -Wl,--rpath-link,/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/lib -Wl,--rpath,/usr/local/bind/lib  BUILD_LIBS=
===>  - Install prefix: /usr/local/bind
===>  Install libtool binary
sudo apt-get update
[sudo] password for spksrc:

hgy59 · 2024-10-15T16:11:27Z

@hgy59 bind compiling tries to install packages ?!?!? I'll try to find where that comes from but in case it ring any bell...

===>  Patching for bind
===>  Configuring for bind
===>  - Configure ARGS: --host=x86_64-pc-linux-gnu --build=i686-pc-linux --prefix=/usr/local/bind --localstatedir=/usr/local/bind/var --enable-full-report BUILD_CC=cc BUILD_CFLAGS=-I/home/spksrc/python-wheels/spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/usr/include -I/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/include -O2 -O2 BUILD_CPPFLAGS=-I/home/spksrc/python-wheels/spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/usr/include -I/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/include -O2 -O2 BUILD_LDFLAGS=-L/home/spksrc/python-wheels/spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/lib -L/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/lib -Wl,--rpath-link,/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/lib -Wl,--rpath,/usr/local/bind/lib  BUILD_LIBS=
===>  - Install prefix: /usr/local/bind
===>  Install libtool binary
sudo apt-get update
[sudo] password for spksrc:

@th0ma7 I will remove this in #6269. It is obsolete since spksrc image on debian 12.

th0ma7 · 2024-10-16T00:11:00Z

@hgy59, @mreid-tt and other @SynoCommunity/developers I'd appreciate having some thoughts on this...

I always kept within spk/python3* examples to build from and test cross-compiled wheels. While it served its purpose it also led to complexifying the overall maintenance of default python makefiles. In hope to simplify this I first thought it would make sense moving all of that under a python-wheels or similar sub-package, sort of placeholder to test wheel building when upgrading python / pip / setuptools and al (around where this PR is atm).

Doing so I thought why not building them all and include other remanants dispersed into other sub spk ...

While it may look neat on paper this becomes a burden to maintain as indivudual spk will evolve over time and this centralized copy won't serve it's purpose any longer...
On the other hand I do believe we need a central location providing clear and up-2-date examples and to test new versions of complex wheels while upgrading python versions.

Then I recall someone mentioning a while back ago, why not having our own wheel repository?

This could avoid maintaining cross-compiled wheels into individual packages + reducing package size as it would then download from our wheel repository at install time.
This centralized python-wheels could potentially meet that purpose.
While neat on paper I wonder what effect would lead upgrading dependent cross/* existing wheels ...

Thoughts on this would be much appreciated.

Also on my TODO (to which babysteps may be best):

make available a python 3.13 package (undergoing)
fix CMake based wheel building (requires sort of autodetection to allow passing proper parameters to at build-time)
fix meson based wheel building (similar issue to CMake type)
decouple crossenv from cross/python3* makefiles
allow either on-demand OR legacy+latest crossenv (for compatibility reasons)

mreid-tt · 2024-10-17T09:25:52Z

@th0ma7, I’m not very experienced with Python builds, so I had to familiarize myself with the basics (e.g., using Real Python's guide on wheels). Here are some initial thoughts:

Moving test code examples to "build from and test cross-compiled wheels" seems like a solid idea, aligning well with the purpose of the demoservice and demowebservice packages.
Setting up and maintaining our own wheel repository appears to be a significant long-term commitment.

I might be missing some key points due to my limited background, so I'd appreciate more details. For instance, you mentioned integrating other "remnants" scattered across sub SPKs. What exactly are these remnants, and what benefits would centralizing them bring? Are they also related to test code?

Regarding the internal wheel repository, is it a matter of PyPI's offerings being insufficient? Are there commonly missing platform-specific wheels for Synology hardware? Would it be feasible to advocate for the Python Package Index to include the wheels we need, perhaps by reaching out to the relevant projects and requesting support?

I might not fully grasp the situation, but I’m keen to understand the challenges better and contribute more meaningful suggestions.

th0ma7 · 2024-10-19T15:40:54Z

I’m not very experienced with Python builds, so I had to familiarize myself with the basics (e.g., using Real Python's guide on wheels). Here are some initial thoughts:

Moving test code examples to "build from and test cross-compiled wheels" seems like a solid idea, aligning well with the purpose of the demoservice and demowebservice packages.

Indeed. Package could even be renamed similarly.

Setting up and maintaining our own wheel repository appears to be a significant long-term commitment.

Not really. That would mostly be just a web page listing our wheels, that you can query through json to list our pre-compiled wheel packages. From there in our requirements.txt file provided with each package, we add a URL to fetch from. Then when wheels gets installed at package installation, it can download them from our source, similarly to using pypi.

I might be missing some key points due to my limited background, so I'd appreciate more details. For instance, you mentioned integrating other "remnants" scattered across sub SPKs. What exactly are these remnants, and what benefits would centralizing them bring? Are they also related to test code?

Regarding the internal wheel repository, is it a matter of PyPI's offerings being insufficient? Are there commonly missing platform-specific wheels for Synology hardware? Would it be feasible to advocate for the Python Package Index to include the wheels we need, perhaps by reaching out to the relevant projects and requesting support?

Issue is, pypi provides tons of pre-compiled wheels but ppc, armv5, (and often armv7 and even aarch64) are always missing. Therefore, on our NAS at installation time pip will only find the source of the requested wheels and try to compile them locally... and failing miserably as our Synology NAS does not provide a pre-installed build system toolchain and we're not providing the include files of the various dependencies.

Therefore when we build our spk package using our spksrc framework, we pull source packages from pypi usng pip, then cross-compile them and generate a resulting *.whl file. This wheel file gets packaged withing the spk so at package installation installation time it is being fetched by pip after failing to find a valid source online.

I might not fully grasp the situation, but I’m keen to understand the challenges better and contribute more meaningful suggestions.

The challenge is that the python wheel build system is undergoing a lot of changes currently and impacting the overall wheel building approaches. As pypi modules documentation and maintenance varies from one to the next, this ends-up breaking things all over the place.

So question is, would it be easier to have a single location to manage all wheel cross-compiling, providing numerous versions online to ease package management? I've been thinking of this a little more and still unsure, further as this would probably mean statically linking wheels so it is compatible with any installation OR using rpath so dependencies are available from that python-wheel package for instance.

Safihre · 2024-10-20T06:42:41Z

It makes sense to have our own repository. Although it will be quite a lot of work to implement and significantly add to maintenance burden..
I saw there's applications available that allow this, for example https://github.com/pypiserver/pypiserver

…d failure

th0ma7 · 2024-11-14T02:37:06Z

@hgy59 I believe I may now have a feature-wise functional code ready for testing... I'll be away next week (SC24) so cycles until my return may be limited. Although I would much like your opinion on this invasive but in theory fully backward compatible code change.

TL;DR;

Added the ability to create on-demand multiple crossenv (see description above)
In brief, when faced with a legacy or wheels not so well supported, you can now create a mk/crossenv/requirements-<wheel>-<version>.txt OR it falls back to mk/crossenv/requirements-<wheel>.txt if exist a matching crossenv configuration.
Resulting wheel specific crossenv will only be of use for that specific wheel you're trying to compile (unless using fallback mechanism that may work for multiple versions).

Nice addition (from my perspective), ability to re-generate crossenv on demand (see description above for howto).

Besides the non-blocker TODO items, in theory it should be ready for testing and shaking out bugs and/or adjusting proposed strategy. I'm mostly thinking of weird corner-cases, such as the ones found in homeassistant that I hope can be addressed with this (to be tested + wheel specific crossenv configurations to be created).

Let me know if you have a moment to test-bed this, your pair of 👀 would be much appreciated.

Lastly: I did not forgot you with your requirement to join ffmpeg + wheel cross-compiling. With the previous addition of spksrc.ffmpeg.mk, along with this code refactor, it shouldn't be too complext in theory to connect the dots and tweak the code to make this a reality.

EDIT: Looking at github-action output it seems there are still a few rough edges to look for... on my todo list.

hgy59 · 2024-11-14T19:53:53Z

initial analysis:

when building python311 for the first time, the build-crossenv target fails, because some variables are not set, i.e. do not contain version info (I added those variables to the log)

make[3]: Leaving directory '/spksrc/cross/pip'
===>  crossenv wheel packages: pip==24.3.1, setuptools==75.4.0, wheel==0.45.0
===>  crossenv requirements file = /spksrc/mk/crossenv/requirements-default.txt
===>  PYTHON_PKG_VERS =
===>  PYTHON_PKG_NAME = python
===>  PYTHON_PKG_VERS_MAJOR_MINOR = .
mkdir -p /spksrc/spk/python311/work-x64-7.1/Python-/build/lib.linux-x86_64-.
cp -RL /spksrc/native/python/work-native/Python-/build/lib.linux-x86_64-. /spksrc/spk/python311/work-x64-7.1/Python-/build
cp: cannot stat '/spksrc/native/python/work-native/Python-/build/lib.linux-x86_64-.': No such file or directory
make[2]: *** [../../mk/spksrc.crossenv.mk:146: build-crossenv] Error 1
make[2]: Leaving directory '/spksrc/spk/python311'
make[1]: *** [../../mk/spksrc.supported.mk:74: build-arch-x64-7.1] Error 1
make[1]: Leaving directory '/spksrc/spk/python311'

when you call make again, the build-crossenv target is not built again (this looks like a missing .-response file)

but after call make crossenvclean

the the variables are set (and build finally succeeds):

make[3]: Leaving directory '/spksrc/cross/pip'
mkdir -p /spksrc/spk/python311/work-x64-7.1/crossenv-default/build
===>  crossenv wheel packages: pip==24.3.1, setuptools==75.4.0, wheel==0.45.0
===>  crossenv requirements file = /spksrc/mk/crossenv/requirements-default.txt
===>  PYTHON_PKG_VERS = 3.11.10
===>  PYTHON_PKG_NAME = python311
===>  PYTHON_PKG_VERS_MAJOR_MINOR = 3.11
mkdir -p /spksrc/spk/python311/work-x64-7.1/Python-3.11.10/build/lib.linux-x86_64-3.11
cp -RL /spksrc/native/python311/work-native/Python-3.11.10/build/lib.linux-x86_64-3.11 /spksrc/spk/python311/work-x64-7.1/Python-3.11.10/build
/spksrc/native/python311/work-native/install/usr/local/bin/python3 -m crossenv /spksrc/spk/python311/work-x64-7.1/install/var/packages/python311/target/bin/python3.11 --cc /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-gcc --cxx /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-c++ --ar /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-ar --sysroot /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root --env LIBRARY_PATH= --manylinux manylinux2014 /spksrc/spk/python311/work-x64-7.1/crossenv-default
===>  Setting default crossenv /spksrc/spk/python311/work-x64-7.1/crossenv-default
===>  ln -sf crossenv-default /spksrc/spk/python311/work-x64-7.1/crossenv
...

hgy59 · 2024-11-14T20:13:01Z

some more details to the analysis above

all lines from build log starting with ===> (i.e. @$(MSG)) without dependent packages:

	Zeile  8633: ===>  Extracting for python311
	Zeile  8635: ===>  Patching for python311
	Zeile  8644: ===>  Configuring for python311
	Zeile  8645: ===>  - Configure ARGS: --host=x86_64-pc-linux-gnu --build=i686-pc-linux --prefix=/var/packages/python311/target --localstatedir=/var/packages/python311/var --enable-shared --without-static-libpython --enable-ipv6 --without-ensurepip --enable-loadable-sqlite-extensions --with-computed-gotos=yes --with-build-python --disable-test-modules --with-lto ac_cv_buggy_getaddrinfo=no ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no ac_cv_have_long_long_format=yes --with-ssl-default-suites=openssl --with-dbmliborder=gdbm:ndbm:bdb --with-system-expat --with-system-ffi
	Zeile  8646: ===>  - Install prefix: /var/packages/python311/target
	Zeile  9394: ===>  Compiling for python311
	Zeile  9943: ===>  Installing for python311
	Zeile 13671: ===>  Correcting pkg-config file lib/pkgconfig/python-3.11-embed.pc
	Zeile 13672: ===>  Correcting pkg-config file lib/pkgconfig/python-3.11.pc
	Zeile 13673: ===>  Correcting pkg-config file lib/pkgconfig/python3-embed.pc
	Zeile 13674: ===>  Correcting pkg-config file lib/pkgconfig/python3.pc
	Zeile 13677: ===>  Downloading files for pip
	Zeile 13678: ===>    File pip-23.2.1.tar.gz already downloaded
	Zeile 13679: ===>  Verifying files for pip
	Zeile 13680: ===>    Checking sha1sum of file pip-23.2.1.tar.gz
	Zeile 13681: ===>    Checking sha256sum of file pip-23.2.1.tar.gz
	Zeile 13682: ===>    Checking md5sum of file pip-23.2.1.tar.gz
	Zeile 13684: ===>  Processing dependencies of pip
	Zeile 13685: ===>  Extracting for pip
	Zeile 13687: ===>  Patching for pip
	Zeile 13688: ===>  Configuring for pip
	Zeile 13689: ===>  - Configure ARGS:
	Zeile 13690: ===>  - Install prefix: /var/packages/python311/target
	Zeile 13696: ===>  crossenv wheel packages: pip==24.3.1, setuptools==75.4.0, wheel==0.45.0
	Zeile 13697: ===>  crossenv requirements file = /spksrc/mk/crossenv/requirements-default.txt
	Zeile 13698: ===>  PYTHON_PKG_VERS = 3.11.10
	Zeile 13699: ===>  PYTHON_PKG_NAME = python311
	Zeile 13700: ===>  PYTHON_PKG_VERS_MAJOR_MINOR = 3.11
	Zeile 13704: ===>  Setting default crossenv /spksrc/spk/python311/work-x64-7.1/crossenv-default
	Zeile 13705: ===>  ln -sf crossenv-default /spksrc/spk/python311/work-x64-7.1/crossenv
	Zeile 13707: ===>  /spksrc/spk/python311/work-x64-7.1/crossenv-default/bin/build-python /spksrc/spk/python311/work-x64-7.1/crossenv-default/build/get-pip.py pip==24.3.1 --no-setuptools --no-wheel --disable-pip-version-check
	Zeile 13717: ===>  /spksrc/spk/python311/work-x64-7.1/crossenv-default/bin/cross-python /spksrc/spk/python311/work-x64-7.1/crossenv-default/build/get-pip.py pip==24.3.1 --no-setuptools --no-wheel --disable-pip-version-check
	Zeile 13727: ===>  /spksrc/spk/python311/work-x64-7.1/crossenv-default/bin/build-pip --disable-pip-version-check install setuptools==75.4.0 wheel==0.45.0
	Zeile 13740: ===>  /spksrc/spk/python311/work-x64-7.1/crossenv-default/bin/cross-pip --disable-pip-version-check install setuptools==75.4.0 wheel==0.45.0
	Zeile 13753: ===>  [/spksrc/spk/python311/work-x64-7.1/crossenv-default] Processing /spksrc/mk/crossenv/requirements-default.txt
	Zeile 13754: ===>  /spksrc/spk/python311/work-x64-7.1/crossenv-default/bin/build-pip --disable-pip-version-check install -r /spksrc/mk/crossenv/requirements-default.txt
	Zeile 13922: ===>  /spksrc/spk/python311/work-x64-7.1/crossenv-default/bin/cross-pip --disable-pip-version-check install -r /spksrc/mk/crossenv/requirements-default.txt
	Zeile 14084: ===>  Package list for /spksrc/spk/python311/work-x64-7.1/crossenv-default:
	Zeile 14085: ===>  /spksrc/spk/python311/work-x64-7.1/crossenv-default/bin/cross-pip list
	Zeile 14147: ===>  Compiling for pip
	Zeile 14148: ===>  CROSSENV: /spksrc/spk/python311/work-x64-7.1/crossenv-default/bin/activate
	Zeile 14150: ===>  Installing for pip
	Zeile 15841: ===>  crossenv wheel packages: pip==24.3.1, setuptools==75.4.0, wheel==0.45.0
	Zeile 15842: ===>  crossenv requirements file = /spksrc/mk/crossenv/requirements-default.txt
	Zeile 15843: ===>  PYTHON_PKG_VERS =
	Zeile 15844: ===>  PYTHON_PKG_NAME = python
	Zeile 15845: ===>  PYTHON_PKG_VERS_MAJOR_MINOR = .

This shows that crossenv-build target is used twice

first for Configuring for pip
second for Installing for pip

the first has correct variables, but the second is missing the python version

This might trigger you...

hgy59 · 2024-11-14T20:26:09Z

@th0ma7 another idea I want to share

When wheels couldn't be built without additional wheels in crossenv (like expandvars to build frozenlist), I was looking in the pyproject.toml file of the source to find the build dependencies.
A mutch more elegant solution would be an automatic population of the crossenv with all build requirements taken from pyproject.toml when the package source has such a file.

th0ma7 · 2024-11-14T20:31:44Z

@th0ma7 another idea I want to share

When wheels couldn't be built without additional wheels in crossenv (like expandvars to build frozenlist), I was looking in the pyproject.toml file of the source to find the build dependencies.
A mutch more elegant solution would be an automatic population of the crossenv with all build requirements taken from pyproject.toml when the package source has such a file.

That would be elegant indeed, and this pr could set the stage as a start to move towards gainin more flexibility with wheel building.

th0ma7 · 2024-11-15T16:50:15Z

@hgy59 now fixed and ready for testing. I also migrated the code to make use of status cookie handling like other pieces of the framework which simplify things a lot.

If by any chances you have a moment to look at py313 cross-compiling... they moved away from setup.py and I believe there may be some work to somehow port the 004-xcompile-paths.patch patch. Help would be welcomed. Note that from testing this switch seems to have started with py312...

hgy59 · 2024-11-15T18:11:08Z

not yet testing...
current build fails for DSM 6.2.4

it fails to install requirements-default.txt as it does not only install the listed modules but additional dependencies.
list of all aditional dependencies that are installed into crossenv see below

For DSM6.2.4 it fails to install msgpack==1.1.0 because it would need additional cflags (use option -std=c99, -std=gnu99, -std=c11 or -std=gnu11 to compile your code)

We must either

use --no-deps to install requirements-default.txt
add msgpack==1.0.5 to requirements-default.txt
install each module in requirements-default.txt with a dedicated call of build-pip/cross-pip with the handling of additional defined flags (as we do for package wheels)

List of additional modules installed into crossenv (those are not listed in requirements-default.txt)

build==1.2.2.post1
CacheControl==0.14.1
certifi==2024.8.30
charset-normalizer==3.4.0
cleo==2.1.0
click==8.1.7
crashtest==0.4.1
distlib==0.3.9
distro==1.9.0
docutils==0.21.2
dulwich==0.21.7
fastjsonschema==2.20.0
filelock==3.16.1
flit_core==3.10.1
idna==3.10
installer==0.7.0
jaraco.classes==3.4.0
jeepney==0.8.0
keyring==24.3.1
more-itertools==10.5.0
msgpack==1.1.0
packaging==24.2
pexpect==4.9.0
pkginfo==1.11.2
platformdirs==4.3.6
poetry-core==1.9.1
poetry-plugin-export==1.8.0
ptyprocess==0.7.0
pycparser==2.22
pyproject_hooks==1.2.0
RapidFuzz==3.10.1
requests==2.32.3
requests-toolbelt==1.0.0
SecretStorage==3.3.3
semantic-version==2.10.0
shellingham==1.5.4
tomli_w==1.1.0
tomlkit==0.13.2
trove-classifiers==2024.10.21.16
urllib3==2.2.3
virtualenv==20.27.1

th0ma7 · 2024-11-15T21:53:52Z

@hgy59 question of managing expectations... This PR really still is a [WIP] and my main focus until now has been on keeping a working python 3.10-3.11 while enhancing the crossenv functionality to have a per-wheel crossenv as needed. Python 3.13 isn't yet fully getting my attraction until I can confirm new crossenv functionality allows re-producing all python based packages.

I was hoping that at this stage it could be tested to confirm wetter this suffice to resolve the immediate build failures we have and allow getting reproducibility back for some of our packages (homeassistant is a good candidate).

not yet testing... current build fails for DSM 6.2.4

it fails to install requirements-default.txt as it does not only install the listed modules but additional dependencies. list of all aditional dependencies that are installed into crossenv see below

Interestingly it's only python 3.13 that fails, and I'm glad it only fails on this as I had other issues previously which now looks solved, at least thru github-action.

For DSM6.2.4 it fails to install msgpack==1.1.0 because it would need additional cflags (use option -std=c99, -std=gnu99, -std=c11 or -std=gnu11 to compile your code)

We must either:

use --no-deps to install requirements-default.txt

The crossenv requires all dependencies to be installed as otherwise it may fail when cross-compile wheels later-on as the build environment would not allow functionalities of said tools to work without its needed dependencies. Also note, this is the exact same behavior as current.

add msgpack==1.0.5 to requirements-default.txt

Currently testing that to see if this helps... Also, on my local branch cross/python313 currenly fails with the following but builds fine using spk/python313 up the the crossenv.

checking for --with-build-python... configure: error: invalid or missing build python binary "python3.13"

I'll have another look at it upon my return (feel free to push fixes if you hapen to have cycles).

install each module in requirements-default.txt with a dedicated call of build-pip/cross-pip with the handling of additional defined flags (as we do for package wheels)

I tried this... and it's really tricky as ordering of install is important. On the other hand that may ease passing -std=c99, -std=gnu99, -std=c11 or -std=gnu11 but even then, we'd need a place to hold such compiler or platform specific crossenv build process configurations beyond current mk/crossenv. An idea could be of using a similar directory functionality as the patch/ directory, something worth investigating later-on.

But you're right, while downgrading msgpack==1.0.5 may solve the immediate issue, we will need extra build options if we intent to maintain backward compatibility with our older platforms (until gcc4 no longer works).

List of additional modules installed into crossenv (those are not listed in requirements-default.txt)

yup, due to the dependency chain as we need to provide all dependencies for cross-compiling to actually work. I find it handy to print the list of crossenv installed wheels to track exactly what is the build environment in use when cross-compiling.

Lastly, my next step is to review the wheel building code to use status cookies and be closer to the remaining of the framework code. I want to divide it such as the following for a start:

spksrc.wheel-download.mk
spksrc.wheel-compile.mk

So then it will become much easier to maintain and add extra functionalities as needed such as meson and cmake toolchain file support, and potentially automating crossenv based on wheel internal information like you suggested.

th0ma7 mentioned this pull request Oct 14, 2024

cross-compiling python 3.13 readline error - unable to find ncurses.pc while ncursesw.pc is available python/cpython#125452

Closed

hgy59 reviewed Oct 14, 2024

View reviewed changes

cross/python313/Makefile Show resolved Hide resolved

SynoCommunity deleted a comment from hacscred Oct 14, 2024

th0ma7 changed the title ~~python311:Dissociate python optional test wheels builds + python313 test drive~~ [WIP] dissociate python optional test wheels builds + python313 Oct 14, 2024

th0ma7 self-assigned this Oct 14, 2024

th0ma7 added the status/work-in-progress label Oct 14, 2024

SynoCommunity deleted a comment from hacscred Oct 15, 2024

th0ma7 mentioned this pull request Oct 15, 2024

readline: Fix readline.pc to depend on ncursesw #6283

Merged

10 tasks

th0ma7 force-pushed the python-wheels branch from d66464e to b66819d Compare October 15, 2024 21:07

th0ma7 mentioned this pull request Oct 16, 2024

Creation of a new spk fails to trigger build #6284

Open

1 task

th0ma7 force-pushed the python-wheels branch from 5a6a413 to ba8d9bd Compare November 3, 2024 21:31

hgy59 mentioned this pull request Nov 10, 2024

rdiff-backup: update and use alternate tmp dir (#4427) #6049

Merged

7 tasks

th0ma7 added 11 commits November 11, 2024 15:03

python311: Remove optional wheels (to be migrated to its own spk)

d61e9d3

python313: Initial commit, fails to build on readline

3a04415

python-wheels: First attempt, to be further developed

3ace2f5

py311: Re-enable lto

4fde552

python313: First working build

81fc40d

python313: Remove pgo optimizations as non-functional in cross-builds

8d79a39

python-wheels: First integration of additional wheels for testing

2c94b2c

mariadb-connector-c: Update from version 3.3.5 to 3.3.11 and fix buil…

2db98e3

…d failure

python313: No longer support armv5 arch due to c++11 support needed

00f488b

spksrc.crossenv.mk: First iteration of on-demand crossenv creation

88e0e7a

spksrc.crossenv.mk: Print crossenv resulting package list

6be5fe0

th0ma7 added 8 commits November 11, 2024 15:05

wheel.mk: Make compatible with spksrc.crossenv.mk

603772e

crossenv.mk: Allow building from PYTHON_PACKAGE dependend spk

0f1d8d1

py310-311: Fix lto vs pgo optimizations

f43b632

borgbackup: Package bump to force build

e95af06

wheel.mk: Use $name-$version using - instead of == for crossenv

5c56f15

python.mk: do not symlink crossenv and python-cc.mk

5b26220

python.mk: Move mandatory PYO3_* variables for rust cross-compiling

ad0902a

crossenv.mk: Fix crossenv creation and pip tool calls

a7a0997

th0ma7 force-pushed the python-wheels branch from bb787f5 to a7a0997 Compare November 11, 2024 15:05

th0ma7 added 10 commits November 13, 2024 01:40

crossenv/requirements-default.txt: Update to latest wheel versions

363b8d6

spk.mk: Add a crossenvclean clean-up option

4018dfc

python.mk + crossenv.mk: Functional versions

5cebad2

python.mk: Further clean-up and removal of duplicate variables

53c13c0

py311-312-312: Update pure requirements to latest versions

df5cf73

py310-311-313: Further clean-up Makefile from unused variables

4aa9b83

py310-311: No longer need to remove test folder

dc6c132

py313: Fix symlink creation

46cb971

crossenv.mk: Include missing OPENSSL_*_DIR

c8f7e4f

spk.mk: Update crossenvclean and add crossenvcleanall

6a41ce0

hgy59 added the framework label Nov 14, 2024

th0ma7 added 2 commits November 15, 2024 14:07

spk.mk: crossenvclean remove cookie file

0abd41b

crossenv: Fix python version and use status cookie

e96693e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] dissociate python optional test wheels builds + python313 #6282

[WIP] dissociate python optional test wheels builds + python313 #6282

th0ma7 commented Oct 14, 2024 •

edited

Loading

th0ma7 commented Oct 14, 2024

th0ma7 commented Oct 15, 2024

hgy59 commented Oct 15, 2024

th0ma7 commented Oct 16, 2024

mreid-tt commented Oct 17, 2024

th0ma7 commented Oct 19, 2024

Safihre commented Oct 20, 2024

th0ma7 commented Nov 14, 2024 •

edited

Loading

hgy59 commented Nov 14, 2024

hgy59 commented Nov 14, 2024

hgy59 commented Nov 14, 2024

th0ma7 commented Nov 14, 2024

th0ma7 commented Nov 15, 2024

hgy59 commented Nov 15, 2024

th0ma7 commented Nov 15, 2024

[WIP] dissociate python optional test wheels builds + python313 #6282

Are you sure you want to change the base?

[WIP] dissociate python optional test wheels builds + python313 #6282

Conversation

th0ma7 commented Oct 14, 2024 • edited Loading

Description

Checklist

Type of change

TODO

th0ma7 commented Oct 14, 2024

th0ma7 commented Oct 15, 2024

hgy59 commented Oct 15, 2024

th0ma7 commented Oct 16, 2024

mreid-tt commented Oct 17, 2024

th0ma7 commented Oct 19, 2024

Safihre commented Oct 20, 2024

th0ma7 commented Nov 14, 2024 • edited Loading

TL;DR;

hgy59 commented Nov 14, 2024

hgy59 commented Nov 14, 2024

hgy59 commented Nov 14, 2024

th0ma7 commented Nov 14, 2024

th0ma7 commented Nov 15, 2024

hgy59 commented Nov 15, 2024

th0ma7 commented Nov 15, 2024

th0ma7 commented Oct 14, 2024 •

edited

Loading

th0ma7 commented Nov 14, 2024 •

edited

Loading