DOC: add documentation about using shared libraries #700

rgommers · 2024-10-27T08:51:27Z

Opening this PR now to get feedback on topics and structure, not ready for detailed review yet.

We get a lot of questions about shared libraries, and this is a tricky thing to get right. So try to document how to use internal libraries as well as link to external shared libraries as well as possible. Subproject-related questions also come up more and more, and there are some extra gotchas here, so treat those as a third "source" of shared (or static) libraries.

Closes Document how shared libraries distributed with the wheel are handled #539
xref Building Windows wheels with libraries fails #525 and ENH: support shared libraries on Windows when explicitly enabled #551 for internal shared libraries on Windows
xref How can I bundle a shared library in a wheel with mesonpy? #410 and Best practice for optionally bundling a library dependency #556 for optionally bundled a shared library

tests/packages/sharedlib-in-package/mypkg/__init__.py

.cirrus.yml

rgommers · 2024-10-28T10:31:44Z

Okay, changed to os.add_dll_directory with some compat notes added. The new test case passes on all Windows configs now except for Cygwin. I don't know what's going on there - I don't think we do anything special in SciPy for Cygwin with the same kind of shared-library-inside-python-package setup. The build completes but at runtime the shared library goes missing:

[1/5] Compiling C object mypkg/cygexamplelib.dll.p/examplelib.c.o
[2/5] Compiling C object mypkg/_example.cpython-39-x86_64-cygwin.dll.p/_examplemod.c.o
[3/5] Linking target mypkg/cygexamplelib.dll
[4/5] Generating symbol file mypkg/cygexamplelib.dll.p/cygexamplelib.dll.symbols
[5/5] Linking target mypkg/_example.cpython-39-x86_64-cygwin.dll
[1/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/.mesonpy-8c69nja6/mypkg/cygexamplelib.dll
[2/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/.mesonpy-8c69nja6/mypkg/libexamplelib.dll.a
[3/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/.mesonpy-8c69nja6/mypkg/_example.cpython-39-x86_64-cygwin.dll
[4/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/.mesonpy-8c69nja6/mypkg/_example.cpython-39-x86_64-cygwin.dll.a
[5/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/mypkg/__init__.py
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pytest-of-runneradmin/pytest-0/mesonpy-test-venv4/lib/python3.9/site-packages/mypkg/__init__.py", line 32, in <module>
    from ._example import example_sum
ImportError: No such file or directory

@DWesl could you perhaps have a look at this? I'm not even sure it's supposed to be working, or it requires support within Cygwin somehow.

dnicolodi · 2024-10-28T10:35:50Z

[1/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/.mesonpy-8c69nja6/mypkg/cygexamplelib.dl

Why does the shared library have this funny name?

dnicolodi · 2024-10-28T10:38:48Z

tests/test_wheel.py

@@ -161,6 +161,13 @@ def test_local_lib(venv, wheel_link_against_local_lib):
    assert int(output) == 3


+@pytest.mark.skipif(MESON_VERSION < (0, 64, 0), reason='Meson version too old')


What does break with older Meson versions?

../meson.build:7:0: ERROR: python.find_installation got unknown keyword arguments "pure"

Oh, this. We work around it in other test packages. I think we should either bump the minimum required Meson version or work around it here too passing the pure argument to the individual functions in the python module. Meson 0.64 was released November 6th 2022, thus almost 2 years ago. Maybe we can update the minimum required version to 0.64 without feeling bad 😃

Yes, I'd prefer to bump to 0.64 rather than continue to use less idiomatic code. Especially since I plan to link to this test package from the docs as a complete and working example.

rgommers · 2024-10-28T10:39:11Z

Why does the shared library have this funny name?

Seems like the fun of building on Windows - with MinGW we get libexamplelib.dll, with MSVC we get examplelib.dll and with Cygwin we get cygexamplelib.dll.

dnicolodi · 2024-10-28T19:28:26Z

tests/packages/sharedlib-in-package/mypkg/__init__.py

+        os.add_dll_directory(basedir)
+
+
+_enable_sharedlib_loading()


Does this really need to define a function and immediately call it? Given that the setup code is reduced to three lines (that can be condensed to two) I think it is clearer to simply have the code directly at module level. Also, I think that it would be nice to have the code actually shown in the documentation page with a Sphinx code include directive. Then the function doc string can be included in the documentation page (I don't think it is easy to have it rendered nicely in the documentation page otherwise).

Makes sense, if it stays at ~2 lines I'll drop the function. .. literalinclude:: is useful too, will do.

For completeness, delvewheel inserts this for external shared libraries that it vendors in:

# start delvewheel patch def _delvewheel_patch_1_8_2(): import os libs_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir, 'numpy.libs')) if os.path.isdir(libs_dir): os.add_dll_directory(libs_dir) _delvewheel_patch_1_8_2() del _delvewheel_patch_1_8_2 # end delvewheel patch

The del is a nice touch if one wants to keep the namespace very clean.

dnicolodi · 2024-10-28T19:39:55Z

Would it be worth to run pyupgrade --py38-plus as part of dropping support for Python 3.7? I've just tried and it upgrades almost all typing annotations (and introduces some bugs about encoding...) thus the noise may be not worth the simplifications.

tests/packages/link-library-in-subproject/meson.build

dnicolodi · 2024-10-28T19:45:33Z

tests/packages/link-library-in-subproject/pyproject.toml

+[project]
+name = 'link-library-in-subproject'
+version = '1.2.3'
+license = 'MIT'


This makes the test dependent on pyproject-metadata 0.9.0 (the test fails on the CI job that tests with pyproject-metadata 0.8.0, which is currently the minimum required version)

Yes, seeing the failure - I'll drop this line then.

In #681 we are updating the minimum required version to 0.9.0. I would kip this.

rgommers · 2024-10-28T19:46:53Z

Would it be worth to run pyupgrade --py38-plus as part of dropping support for Python 3.7? I've just tried and it upgrades almost all typing annotations (and introduces some bugs about encoding...) thus the noise may be not worth the simplifications.

I'll have a look, but if that's useful enough then I'd prefer to do it in a separate PR I think.

DWesl · 2024-10-28T21:03:30Z

Okay, changed to os.add_dll_directory with some compat notes added. The new test case passes on all Windows configs now except for Cygwin. I don't know what's going on there - I don't think we do anything special in SciPy for Cygwin with the same kind of shared-library-inside-python-package setup. The build completes but at runtime the shared library goes missing:

[1/5] Compiling C object mypkg/cygexamplelib.dll.p/examplelib.c.o
[2/5] Compiling C object mypkg/_example.cpython-39-x86_64-cygwin.dll.p/_examplemod.c.o
[3/5] Linking target mypkg/cygexamplelib.dll
[4/5] Generating symbol file mypkg/cygexamplelib.dll.p/cygexamplelib.dll.symbols
[5/5] Linking target mypkg/_example.cpython-39-x86_64-cygwin.dll
[1/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/.mesonpy-8c69nja6/mypkg/cygexamplelib.dll
[2/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/.mesonpy-8c69nja6/mypkg/libexamplelib.dll.a
[3/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/.mesonpy-8c69nja6/mypkg/_example.cpython-39-x86_64-cygwin.dll
[4/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/.mesonpy-8c69nja6/mypkg/_example.cpython-39-x86_64-cygwin.dll.a
[5/5] /cygdrive/d/a/meson-python/meson-python/tests/packages/sharedlib-in-package/mypkg/__init__.py
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pytest-of-runneradmin/pytest-0/mesonpy-test-venv4/lib/python3.9/site-packages/mypkg/__init__.py", line 32, in <module>
    from ._example import example_sum
ImportError: No such file or directory

@DWesl could you perhaps have a look at this? I'm not even sure it's supposed to be working, or it requires support within Cygwin somehow.

Most of the packages I've worked with have only depended on system DLLs/shared libraries, but the reference BLAS and Lapack libraries are in a non-standard location so I have some experience with this from NumPy. The easy things to check are permissions (both shared libraries need execute permissions to work properly) and whether the PATH is set properly to find dependent shared libraries actually linked into the extension module (ldd is the tool you're probably familiar with, though I tend to use cygcheck).

From what I remember of the last time I compiled SciPy on Cygwin a few years back, SciPy doesn't link to itself; dependencies are Python-level or header-only, so this problem wouldn't come up. I haven't run into os.add_dll_directory (in the CPython repo it looks Windows-only and would not be available on Cygwin), I've only ever directly modified PATH outside Python. I wonder whether something like:

if sys.platform == "cygwin":
    def add_dll_directory(path: str):
        os.environ["PATH"] = f"{os.environ['PATH']:s}:{path:s}"
    os.add_dll_directory = add_dll_directory

might help.

tests/packages/sharedlib-in-package/mypkg/__init__.py

rgommers · 2024-10-29T09:34:35Z

Thanks for the input @DWesl!

From what I remember of the last time I compiled SciPy on Cygwin a few years back, SciPy doesn't link to itself; dependencies are Python-level or header-only, so this problem wouldn't come up.

That changed recently, the SciPy 1.14.0 release has a shared library:
https://github.com/scipy/scipy/blob/ea916c6f7f487bd53e98de082649d542cc6106ed/scipy/special/meson.build#L37

I've only ever directly modified PATH outside Python. I wonder whether something like: [...] might help

I'll try but I doubt it, since modifying os.environ shouldn't affect the existing process IIRC.

but the reference BLAS and Lapack libraries are in a non-standard location so I have some experience with this from NumPy.

Is there a patch/repo somewhere for how the numpy package in Cygwin is built?

DWesl · 2024-10-29T11:06:30Z

Thanks for the input @DWesl!

but the reference BLAS and Lapack libraries are in a non-standard location so I have some experience with this from NumPy.

Is there a patch/repo somewhere for how the numpy package in Cygwin is built?

For NumPy:
https://github.com/numpy/numpy/blob/main/.github/workflows/cygwin.yml

For SciPy, most recent I have:
DWesl/scipy#7

That changed recently, the SciPy 1.14.0 release has a shared library

It might still work, depending on what Windows thinks is the current directory for a dlopen call and which extensions load it. I can't test at the moment, since Cygwin doesn't have Python>=3.10 (numpy/numpy#26247)

…te-packages

Note that for Meson versions older than 1.2.0, CI failed with: ``` mesonpy.BuildError: Could not map installation path to an equivalent wheel directory: '{libdir_static}/libexamplelib.a' ``` because the `--skip-subprojects` install option isn't honored. Hence the test skip on older versions.

rgommers · 2024-10-29T17:22:36Z

The Cygwin tests pass now, thanks for the pointers @DWesl. Amending os.environ['PATH'] worked ('LD_LIBRARY_PATH' did not), and is necessary when the shared library is in another directory and also when it is right next to a Python extension module that needs it. For regular Windows, curdir is searched by default so os.add_dll_directory is only necessary when the shared library is elsewhere.

DWesl · 2024-11-04T01:46:10Z

The Cygwin tests pass now, thanks for the pointers @DWesl. Amending os.environ['PATH'] worked ('LD_LIBRARY_PATH' did not),

I think LD_LIBRARY_PATH gets used by dlopen on Cygwin, but Python already has its own logic for that (sys.path), so that's not relevant here.

and is necessary when the shared library is in another directory and also when it is right next to a Python extension module that needs it. For regular Windows, curdir is searched by default so os.add_dll_directory is only necessary when the shared library is elsewhere.

It sounds like Windows does some helpful chdir things/PATH manipulation around its version of dlopen then (LoadDynamicLibraryEx maybe?), that Cygwin (and the lower-level Windows function it passes things off to) does not.
Interesting to know about in case it comes up again.

virtuald · 2024-11-05T05:42:18Z

docs/how-to-guides/shared-libraries.rst

+``delvewheel`` in order to make the built Python package usable locally.
+
+
+Publishing wheels which depend on external shared libraries


There's no mention here of the topic of linking to libraries contained in other wheels. Even if it's not explained in the final draft, it should probably be mentioned and TODO added.

Thanks for the suggestion @virtuald. I'm not quite sure I want to add that topic though. Just to clarify, did you mean (a) an actual shared library contained in a wheel (a la scipy-openblas32), or (b) a Python package that exports a C API (a la NumPy's C API)?

I think (b) doesn't really fit topic-wise, and (a) isn't recommended, since it requires a bunch of manual workarounds to get to work at all, and then there's no good way to deal with versioning and upgrades.

I mean (a), as I think when you start discussing 'external' libraries eventually that thought comes up, so it should be given at least some mention. As I said, even if it's just a footnote that says "you probably shouldn't do this" or something else to that effect, I think that would be appropriate enough for now.

Makes sense, thanks - I'll add a note like that.

rgommers added the documentation Improvements or additions to documentation label Oct 27, 2024

rgommers force-pushed the doc-sharedlibs branch from 2dd3221 to d254472 Compare October 27, 2024 19:15

dnicolodi reviewed Oct 27, 2024

View reviewed changes

tests/packages/sharedlib-in-package/mypkg/__init__.py Outdated Show resolved Hide resolved

rgommers force-pushed the doc-sharedlibs branch 2 times, most recently from 44f372e to 89dbcc7 Compare October 28, 2024 10:14

dnicolodi reviewed Oct 28, 2024

View reviewed changes

.cirrus.yml Outdated Show resolved Hide resolved

dnicolodi reviewed Oct 28, 2024

View reviewed changes

rgommers force-pushed the doc-sharedlibs branch 2 times, most recently from ddaea0f to f95d160 Compare October 28, 2024 19:22

dnicolodi reviewed Oct 28, 2024

View reviewed changes

tests/packages/link-library-in-subproject/meson.build Outdated Show resolved Hide resolved

dnicolodi reviewed Oct 28, 2024

View reviewed changes

DWesl reviewed Oct 28, 2024

View reviewed changes

tests/packages/sharedlib-in-package/mypkg/__init__.py Show resolved Hide resolved

rgommers mentioned this pull request Oct 29, 2024

MAINT: bump minimum required Meson version to 0.64.0 #701

Merged

rgommers added 4 commits October 29, 2024 18:05

WIP: DOC: add documentation about using shared libraries

fd1b476

MAINT: drop Python 3.7 support

35348fc

TST: add test package with internal shared libraries, installed in si…

6d7de48

…te-packages

rgommers force-pushed the doc-sharedlibs branch from f95d160 to 83e194b Compare October 29, 2024 17:10

rgommers mentioned this pull request Oct 31, 2024

binutils 2.43.50: Segmentation fault in test_local_lib #698

Open

virtuald reviewed Nov 5, 2024

View reviewed changes

virtuald mentioned this pull request Nov 11, 2024

Add explicit support for linking to/from external wheels virtuald/hatch-meson#8

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DOC: add documentation about using shared libraries #700

DOC: add documentation about using shared libraries #700

rgommers commented Oct 27, 2024

rgommers commented Oct 28, 2024

dnicolodi commented Oct 28, 2024 •

edited

Loading

dnicolodi Oct 28, 2024

rgommers Oct 28, 2024

dnicolodi Oct 28, 2024

rgommers Oct 28, 2024

rgommers commented Oct 28, 2024

dnicolodi Oct 28, 2024

rgommers Oct 29, 2024

dnicolodi commented Oct 28, 2024

dnicolodi Oct 28, 2024

rgommers Oct 28, 2024

dnicolodi Oct 28, 2024

rgommers commented Oct 28, 2024

DWesl commented Oct 28, 2024

rgommers commented Oct 29, 2024

DWesl commented Oct 29, 2024

rgommers commented Oct 29, 2024

DWesl commented Nov 4, 2024

virtuald Nov 5, 2024

rgommers Nov 5, 2024

virtuald Nov 5, 2024

rgommers Nov 5, 2024

		@@ -161,6 +161,13 @@ def test_local_lib(venv, wheel_link_against_local_lib):
		assert int(output) == 3


		@pytest.mark.skipif(MESON_VERSION < (0, 64, 0), reason='Meson version too old')

		``delvewheel`` in order to make the built Python package usable locally.


		Publishing wheels which depend on external shared libraries

		os.add_dll_directory(basedir)


		_enable_sharedlib_loading()

DOC: add documentation about using shared libraries #700

Are you sure you want to change the base?

DOC: add documentation about using shared libraries #700

Conversation

rgommers commented Oct 27, 2024

rgommers commented Oct 28, 2024

dnicolodi commented Oct 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rgommers commented Oct 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dnicolodi commented Oct 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rgommers commented Oct 28, 2024

DWesl commented Oct 28, 2024

rgommers commented Oct 29, 2024

DWesl commented Oct 29, 2024

rgommers commented Oct 29, 2024

DWesl commented Nov 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dnicolodi commented Oct 28, 2024 •

edited

Loading