Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide python executable compiled with enable-shared #595

Open
LunarLanding opened this issue Nov 10, 2022 · 4 comments
Open

Provide python executable compiled with enable-shared #595

LunarLanding opened this issue Nov 10, 2022 · 4 comments
Labels

Comments

@LunarLanding
Copy link

LunarLanding commented Nov 10, 2022

Comment:

Related: #222 (comment) .
My use-case: lammps/lammps#3520

This could be provided as a subpackage, say python-shared, and install a python.VERSION_shared that the user could symlink or put in front of $PATH, as needed.

I would have built it myself but I am a bit lost since building this recipe with enable-shared is not being easy.

The usual ways of debugging a failing conda build do not work with this package: conda debug ends up giving me an environment where there's no script to activate the environment and no build script either, perhaps because the recipe has multiple outputs and conda debug does not know what to do with it.

So far had to remove the libpython-static output, make the recipe a single output recipe, move build_base.sh to build.sh, edit out script:build.sh, comment out rm .../libpython(...)$VERSION(...).a in build_shared, and add f to ln -sf ../../libpython${VERABI}.a libpython${VERABI}.a in build_static.

The last hurdle is that after building, conda build packages it, where it alters the paths that libraries/execs refer too ( I'm not very knowledgeable about this) and it fails with this message, which I guess has to do with signing in osx-arm64.
PS: fixed it by editing conda-builld/.../post.py conda/conda-build#4392 (comment)

Tests do not pass (tkinter something), but I installed the package anyways and for my use-case it seems to function
PS: does not. conda_build_config.yaml included in the repo overrode the one I had in my home folder, so it did not pick up on the openssl==1.1 constraint and then when I replaced the python in my installation with the shared one it complained it could not find libssl.3.dylib.

After fixing that, for some reason 'libffi.7.dylib' was required by 'import _ctypes', which I fixed in the install environment by symlinking 'libffi.8.dylib' to 'libffi.7.dylib'.

@jakirkham
Copy link
Member

All of the symbols are in the python executable. One can just link to that. Is there a reason that doesn't work for your use case?

@LunarLanding
Copy link
Author

LunarLanding commented Nov 11, 2022

@jakirkham Thanks for the suggestion. I think currently the library I'm trying to use links against libpython. It does so somehow using a cmake script. Do you know if I need to recompile or if I can patch the library to point to the python executable?
PS: found a description here .
What makes my use-case a bit more complex is that I am compiling/running on osx-arm64.

@jakirkham
Copy link
Member

Sometimes build scripts can be tricked by creating a symlink from libpython to the python executable. It is a bit of a hack, but may get things working quickly. Would just make sure to delete the symlink at the the end of the build (so it doesn't accidentally wind up in the package and cause issues)

@LunarLanding
Copy link
Author

At least on my platform (OSX arm64), it fails.

Build script:

# LIBPATH=.../libpython3.10.dylib
# EXECPATH=.../python3.10
mv $LIBPATH $LIBPATH.tmp
trap "mv $LIBPATH.tmp $LIBPATH" EXIT
ln -s $EXECPATH $LIBPATH

...
(cmake)

Output:

ld: warning: -pie being ignored. It is only used when linking a main executable
ld: can't link with a main executable file '$LIBPYTHON'
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)

copybara-service bot pushed a commit to google-deepmind/mujoco that referenced this issue Mar 17, 2023
- Find Python functions in the interpreter binary itself rather than trying to find libpython.dylib, because this doesn't exist when using Conda. (conda-forge/python-feedstock#595 (comment))

- Increase the stack size of both the Python main thread and the OS main thread to 16MiB, to match what Python normally uses on macOS. (https://github.com/python/cpython/blob/3.11/configure#L11038)

- Check whether dlfcn and pthread function calls actually succeeds. Emit error messages and exit on failure rather than continuing the program (which generally leads to segfaults).

- Fix pixelated shadow in the icon.

PiperOrigin-RevId: 517390522
Change-Id: Ib91679b4baa63bac5f6b5893e634712d72161ea5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants