Recommended build environment for Loom now with Python3?

[ships]

Hi friends, I am trying to build a reproducible environment using both Docker and Nix that can run the latest Loom and Distributions.

I have had problems recently with compiling distributions on current head of main on ubuntu 20.04 because the GCC version is too new, but I see some references to later python versions like 3.10 that are not supported until 20.04 or later.

While I still have some threads to chase down in my build issues, I would love to know, what sort of compile and build environments and targets are you testing on internally ?

[ships]

For example, the current build issue appears to come with Python scripts happening as part of make install_cy inside distributions, when on python 3.8 with virtualenv in a clean Ubuntu 20.04 container.

The virtualenv was created with python3 -m venv /.venv.

#16 43.43 pip install -r requirements.txt
#16 43.88 Requirement already satisfied: setuptools>=2.2 in /.venv/lib/python3.8/site-packages (from -r requirements.txt (line 1)) (44.0.0)
#16 44.27 Collecting numpy>=1.6.1 (from -r requirements.txt (line 2))
#16 44.36   Downloading numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB)
#16 44.53 Collecting scipy>=0.9.0 (from -r requirements.txt (line 3))
#16 44.55   Downloading scipy-1.10.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (58 kB)
#16 44.58      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.9/58.9 kB 2.3 MB/s eta 0:00:00
#16 44.96 Collecting cython>=0.20.1 (from -r requirements.txt (line 4))
#16 44.98   Downloading Cython-3.0.10-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.2 kB)
#16 45.10 Collecting scikit-learn (from -r requirements.txt (line 5))
#16 45.12   Downloading scikit_learn-1.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)
#16 45.15 Collecting goftests (from -r requirements.txt (line 6))
#16 45.55   Downloading goftests-0.3.0-py2.py3-none-any.whl.metadata (5.9 kB)
#16 45.69 Collecting simplejson (from -r requirements.txt (line 7))
#16 45.71   Downloading simplejson-3.19.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.1 kB)
#16 45.94 Collecting protobuf (from -r requirements.txt (line 8))
#16 45.96   Downloading protobuf-5.26.1-cp37-abi3-manylinux2014_x86_64.whl.metadata (592 bytes)
#16 45.99 Collecting parsable (from -r requirements.txt (line 9))
#16 46.01   Downloading parsable-0.3.1-py3-none-any.whl.metadata (3.2 kB)
#16 46.04 Collecting pyflakes (from -r requirements.txt (line 10))
#16 46.06   Downloading pyflakes-3.2.0-py2.py3-none-any.whl.metadata (3.5 kB)
#16 46.11 Collecting pep8 (from -r requirements.txt (line 11))
#16 46.12   Downloading pep8-1.7.1-py2.py3-none-any.whl.metadata (22 kB)
#16 46.16 Collecting nose (from -r requirements.txt (line 12))
#16 46.17   Downloading nose-1.3.7-py3-none-any.whl.metadata (1.7 kB)
#16 46.21 Collecting cpplint (from -r requirements.txt (line 13))
#16 46.22   Downloading cpplint-1.6.1-py3-none-any.whl.metadata (4.5 kB)
#16 46.32 Collecting joblib>=1.1.1 (from scikit-learn->-r requirements.txt (line 5))
#16 46.34   Downloading joblib-1.3.2-py3-none-any.whl.metadata (5.4 kB)
#16 46.36 Collecting threadpoolctl>=2.0.0 (from scikit-learn->-r requirements.txt (line 5))
#16 46.38   Downloading threadpoolctl-3.4.0-py3-none-any.whl.metadata (13 kB)
#16 46.44 Downloading numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
#16 53.77    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 2.4 MB/s eta 0:00:00
#16 53.79 Downloading scipy-1.10.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
#16 69.13    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 2.2 MB/s eta 0:00:00
#16 69.18 Downloading Cython-3.0.10-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
#16 71.07    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 2.0 MB/s eta 0:00:00
#16 71.15 Downloading scikit_learn-1.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.1 MB)
#16 77.29    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.1/11.1 MB 1.8 MB/s eta 0:00:00
#16 77.38 Downloading goftests-0.3.0-py2.py3-none-any.whl (15 kB)
#16 77.50 Downloading simplejson-3.19.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (136 kB)
#16 77.60    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.6/136.6 kB 1.5 MB/s eta 0:00:00
#16 77.70 Downloading protobuf-5.26.1-cp37-abi3-manylinux2014_x86_64.whl (302 kB)
#16 77.87    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 302.8/302.8 kB 1.8 MB/s eta 0:00:00
#16 77.96 Downloading parsable-0.3.1-py3-none-any.whl (4.5 kB)
#16 78.03 Downloading pyflakes-3.2.0-py2.py3-none-any.whl (62 kB)
#16 78.20    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.7/62.7 kB 380.0 kB/s eta 0:00:00
#16 78.37 Downloading pep8-1.7.1-py2.py3-none-any.whl (41 kB)
#16 78.39    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.5/41.5 kB 4.4 MB/s eta 0:00:00
#16 78.42 Downloading nose-1.3.7-py3-none-any.whl (154 kB)
#16 78.67    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 154.7/154.7 kB 592.8 kB/s eta 0:00:00
#16 78.88 Downloading cpplint-1.6.1-py3-none-any.whl (77 kB)
#16 78.91    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.3/77.3 kB 2.3 MB/s eta 0:00:00
#16 79.25 Downloading joblib-1.3.2-py3-none-any.whl (302 kB)
#16 81.22    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 302.2/302.2 kB 149.3 kB/s eta 0:00:00
#16 81.39 Downloading threadpoolctl-3.4.0-py3-none-any.whl (17 kB)
#16 81.67 Installing collected packages: pep8, parsable, nose, cpplint, threadpoolctl, simplejson, pyflakes, protobuf, numpy, joblib, cython, scipy, scikit-learn, goftests
#16 86.85 Successfully installed cpplint-1.6.1 cython-3.0.10 goftests-0.3.0 joblib-1.3.2 nose-1.3.7 numpy-1.24.4 parsable-0.3.1 pep8-1.7.1 protobuf-5.26.1 pyflakes-3.2.0 scikit-learn-1.3.2 scipy-1.10.1 simplejson-3.19.2 threadpoolctl-3.4.0
#16 87.44 LIBRARY_PATH=:/.venv/lib/ pip install --upgrade .
#16 87.86 Processing /distributions
#16 87.86   Installing build dependencies: started
#16 89.48   Installing build dependencies: finished with status 'done'
#16 89.48   Getting requirements to build wheel: started
#16 89.66   Getting requirements to build wheel: finished with status 'error'
#16 89.67   error: subprocess-exited-with-error
#16 89.67
#16 89.67   × Getting requirements to build wheel did not run successfully.
#16 89.67   │ exit code: 1
#16 89.67   ╰─> [17 lines of output]
#16 89.67       Traceback (most recent call last):
#16 89.67         File "/.venv/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
#16 89.67           main()
#16 89.67         File "/.venv/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
#16 89.67           json_out['return_val'] = hook(**hook_input['kwargs'])
#16 89.67         File "/.venv/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
#16 89.67           return hook(config_settings)
#16 89.67         File "/tmp/pip-build-env-k19xz1o4/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
#16 89.67           return self._get_build_requires(config_settings, requirements=['wheel'])
#16 89.67         File "/tmp/pip-build-env-k19xz1o4/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
#16 89.67           self.run_setup()
#16 89.67         File "/tmp/pip-build-env-k19xz1o4/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 487, in run_setup
#16 89.67           super().run_setup(setup_script=setup_script)
#16 89.67         File "/tmp/pip-build-env-k19xz1o4/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 311, in run_setup
#16 89.67           exec(code, locals())
#16 89.67         File "<string>", line 31, in <module>
#16 89.67       ModuleNotFoundError: No module named 'numpy'
#16 89.67       [end of output]

[ships]

Separately, we are trying to compile for ARM64 and get a variety of issues:

• -msse4.1 flag should only be passed when supported
• x86_intrin.h
• _mm_ prefixed intrinsics

Is it expected that this library only work on AMD / x86_64 ?

[fritzo]

Hi @ships, I'd love for loom to have reproducible modern builds. Recently @emilyfertig has been updating loom & distributions, and is probably your best contact. I'm happy to review PRs, push releases, and answer questions.

Re: aarch64, this may be a more significant effort. The scariest issue is that loom uses some lock-free data structures internally, and I believe existing code relies on x86 semantics. It might be possible to simply disable lock-free stuff if not on x86, but that's just a guess.