To provide wheels files for Windows, Linux, ...

[mantielero]

I am unable to install this package in Windows. It would be nice if binary installation packages were produced. Such as wheels files.

[fedelibre]

Perhaps PR #41 may solve this? I haven't tested it.
You can try:

pip install git+https://github.com/mitya57/python-poppler-qt5.git@sip5

[mantielero]

It is not working for me. I will post my issue in that thread.

Nonetheless, it would still be desirable to have a wheels for installation. There are many issues about the installation from those of us who are not that savvy.

[robin25]

Same for me, i'm not able to install on windows.

As I installed only PyQT5 on my system I haven't any qmake installed.

[fedelibre]

You should first run these commands:

pip install --upgrade pip
pip install --upgrade sip

then try installing python-poppler-qt5:

pip install git+https://github.com/mitya57/python-poppler-qt5.git@sip5

If you get an error, check #41 and report the problem if it's not been reported by others already.

[fedelibre]

@robin25 Sorry, I should not have pointed to that MR. It does not provide wheels for Windows, it just allows to build this module with sip v5 (instead of v4).

[robin25]

Hey @fedelibre anyway thank you for your help. I've tried but as you said it not provide wheels. I'm always looking for a workaround.

[Knightowl3128]

@robin25 @mantielero Were you able to install it finally?

[fedelibre]

@robin25 After reading this comment I've just realized that now we can build python-poppler-qt5 using the sip-* commands (which read the pyproject.toml file).

Building a wheel for Linux seems as easy as running sip-wheel.
Here's the built file:

$ ls -l *.whl
-rw-r--r--. 1 fede fede 178726 Oct 12 23:27 python_poppler_qt5-21.1.0-cp310-cp310-manylinux2014_x86_64.whl

Perhaps running the same command on a Windows machine would allow to have a working wheel for Windows users? (to be uploaded on PyPI)

[fedelibre]

I tried using Github Actions and cibuildwheel to build wheels (initially for Linux, but can be done for Mac and Windows as well).

I'm using manylinux_2_24, the supported image based on Debian 9. If you compare it with the other manylinux images, it has the advantage of having Qt5 packages, but the big disadvantage of having a very old gcc compiler (6.3). In fact I think that the build is failing because of that, see this log:

https://github.com/fedelibre/python-poppler-qt5/runs/4015398642

There's an open issue in manylinux repository about upgrading GCC version in manylinux_2_24.

@mitya57 @bnavigator maybe you have some suggestions?

[mitya57]

Hi @fedelibre, and sorry for late response.

The problem is old Poppler version. You have Poppler 0.48, as can be seen in the log:

Get:114 http://security.debian.org/debian-security stretch/updates/main amd64 libpoppler-dev amd64 0.48.0-2+deb9u4 [774 kB]

But FormFieldText::setFontSize method was added in Poppler 0.60:
https://gitlab.freedesktop.org/poppler/poppler/-/commit/dfcf997e6fbca31d

You may wrap use of these new methods in %If(POPPLER_V0_60_0 -) … %End blocks, it should help.

It seems like no one else tested building with such an old version, so probably there will be other places where you will need to do the same.

[jeanas]

I'm looking into this, and I'm not quite sure how versioning is supposed to work for our wheels. See question: https://discuss.python.org/t/packaging-a-c-extension-with-a-dependency-on-another-c-extension/24462

[fedelibre]

@jeanas See my cibuildwheel branch. I managed to set up Github actions to build wheels for Linux and Mac. I had problems with Windows and eventually gave up.

[fedelibre]

Don't know if it answers your question, but if you download the artifact of this successful build you'll see the following names used by cibuildwheel:

[fede@fedora artifact]$ ls
python_poppler_qt5-21.3.0-cp310-cp310-macosx_10_13_x86_64.whl
python_poppler_qt5-21.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
python_poppler_qt5-21.3.0-cp311-cp311-macosx_10_13_x86_64.whl
python_poppler_qt5-21.3.0-cp311-cp311-manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
python_poppler_qt5-21.3.0-cp37-cp37m-macosx_10_13_x86_64.whl
python_poppler_qt5-21.3.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
python_poppler_qt5-21.3.0-cp38-cp38-macosx_10_13_x86_64.whl
python_poppler_qt5-21.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl
python_poppler_qt5-21.3.0-cp39-cp39-macosx_10_13_x86_64.whl
python_poppler_qt5-21.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl

[jeanas]

Nice, I didn't know you had tried this in the past.

Those file names are precisely what my question is about. I don't see a place in those names for the PyQt5 version. It seems to me that using these wheels can only work if you install a specific PyQt5 version, but if they were uploaded on PyPI, pip-installing them would not work unless the PyQt5 version chosen by pip happened to be the same or ABI-compatible. We'd need to pin the PyQt5 version in pyproject.toml. This would work, but it's usually considered bad practice for a library to have pinned dependencies (because it creates dependency hells).

[jeanas]

(However, for Frescobaldi itself, it wouldn't be a problem.)

[jeanas]

Also https://www.riverbankcomputing.com/pipermail/pyqt/2023-March/045177.html

[jeanas]

Nice, I didn't know you had tried this in the past.

Well, I should have read the rest of the thread less hastily 😊

[jeanas]

@fedelibre I'm trying my hands on this based on your work. We'll see how far I go.

[fedelibre]

👍 good luck!

[jeanas]

Well, I'm stuck at the same point as you, namely Windows.

I'm looking into Conda, maybe it will be ultimately easier to just ship python-poppler-qt5 as a conda package, even if this doesn't correspond to the initial goal of building wheel files for the PyPA toolchain (i.e., pip, setuptools, etc.). Or maybe it will work to build wheels in a Conda environment and later reuse them in a non-Conda environment, but I'm not 100% sure.

[jeanas]

My WIP can be found in the wheels branch of my fork. As you can see from https://github.com/jeanas/python-poppler-qt5/actions/runs/4337912590, after two days of work and lots of fun (hint: one of these two things is a joke), it's starting to look like something, though there are still sporadic failures that I need to debug.

One thing I'm not really understanding is how one is supposed to build shared libraries that will work for a wide range of macOS and Windows versions. For Linux, it's easy: just build on old enough manylinux, to have an old enough glibc. But for macOS and Windows, I don't know. Does anyone have insights on this?

[fedelibre]

One thing I'm not really understanding is how one is supposed to build shared libraries that will work for a wide range of macOS and Windows versions. For Linux, it's easy: just build on old enough manylinux, to have an old enough glibc. But for macOS and Windows, I don't know. Does anyone have insights on this?

I don't. Is this helpful?

Checking what other projects using cibuildwheel do might give us some hints. I started with scikit-learn:

• https://github.com/scikit-learn/scikit-learn/tree/main/build_tools/github
• https://pypi.org/project/scikit-learn/#files

but it's quite complex

[jeanas]

Thanks, Federico, I'll take a look.

Another problem and question: https://www.riverbankcomputing.com/pipermail/pyqt/2023-March/045186.html

[fedelibre]

Another problem and question: https://www.riverbankcomputing.com/pipermail/pyqt/2023-March/045186.html

After a quick search I saw people recommending auditwheel for linux and delocate for Mac. No ready solution for Windows. Maybe not compatible with the manylinux 2_17 version you're using?

[jeanas]

Yeah, it can be added to the wheels a posteriori, but I'd prefer to avoid that since, as you've found out, it requires OS-dependent tools.

On the PyQt mailing list, the PyQt / SIP author replied with a config setting that we should be able to set in project.py to make this happen. I just need to figure out how to get the path to the shared lib in conda.

[jeanas]

This is an absolute nightmare.

For the wheel to work, we don't just need libpoppler and libpoppler-qt5, but also lots of libraries like libfontconfig, libfreetype, libpng, libharfbuzz, libjpeg, etc. Not sure how I missed the other time that these were linked dynamically as well. I don't know how to reliably get the list of needed libraries; maybe parsing the ldd output, though it sounds fragile. I don't even know what the Windows equivalent would be.

Or we can try to build a static poppler-qt5 that only links dynamically to Qt libs and statically to all the rest, but it means we have to compile Poppler by ourselves. Same here, I don't know what the CMake flags / trickery for this would be.

Last option, we can give up on wheels entirely and provide a Conda package. Meaning that end users will have to use Conda instead of pip. But it's starting to look like the most attractive option, TBH.

[fedelibre]

I don't know how to reliably get the list of needed libraries; maybe parsing the ldd output, though it sounds fragile. I don't even know what the Windows equivalent would be.

ldd for Windows is available through Cygwin (and Github Actions). See:
https://github.com/egor-tensin/setup-cygwin

[jeanas]

https://discuss.python.org/t/wheel-depending-on-shared-library-from-another-wheel/26456

[jeanas]

Here is a path forward.

For one thing, we pin the PyQt5 version. The wheels will be incompatible with other PyQt5 versions anyway. We guarantee that people won't get errors about missing symbols or such during dynamic linking at module load. Admittedly, the wheels might be compatible with higher PyQt5 versions than the one they were built for (within certain limits).

People who want python-poppler-qt5 with a different version of PyQt will not be more in trouble than they are today, with the lack of any binary distribution.

For another thing, I've found it complicated to use auditwheel/delocate/delvewheel to bundle external libs into wheels because we don't want the PyQt ones, and especially delvewheel doesn't have any option to exclude one library. So, let's just list them manually in the project.py file.

That is well, but what happens for the popplerqt5 dynamic lib to find the Qt ones? According to the d.p.o. thread linked above, the main portable solution for making popplerqt5 find Qt is to put both in the same directory. It's actually not as bad as I thought. QScintilla seems to do this. We can make popplerqt5 installed under PyQt5._popplerqt5, and ship a popplerqt5.py module that just does from PyQt5._popplerqt5 import *.

A weak point is that this might not work if PyQt5 and python-poppler-qt5 are installed in different locations. However, the common case of this is that PyQt5 is installed through the distro and python-poppler-qt5 is not. In this case, all is still fine. A well-behaved dynamic linker should search in /usr/lib if it doesn't find a library in the same directory as the one that has a dependency.

The main case where this fails is when PyQt5 is installed in the user site directory (pip install PyQt5), and python-poppler-qt5 is installed into a virtual environment that has access to user site packages (python -m venv --system-site-packages). I would expect that to be pretty rare.

(Also, installing PyQt5 in the user site package dir is a bad idea anyway, since it can break system apps.)

This... is suboptimal on a few points and makes several compromises, but I hope it will work decently. Feel free to comment, of course.

[jeanas]

Houston, we have a problem.

I was able to successfully build and run Linux wheels as described above. They mostly work, but there are Fontconfig errors. The reason is that conda-forge patches Fontconfig to load config files from the conda installation. That doesn't work outside of conda.

If we let the package load the system libfontconfig, there could be errors due to the older ABI.

:/

At this point, I'm going to try to submit a python-poppler-qt5 package on conda-forge, and if that works, abandon the idea of building wheels.

[jeanas]

See conda-forge/staged-recipes#22792

@fedelibre If I manage to make it work, do you want to be listed as a maintainer on the conda-forge package?

[fedelibre]

@fedelibre If I manage to make it work, do you want to be listed as a maintainer on the conda-forge package?

Ok, thanks

[jeanas]

The conda-forge package unfortunately turned out to be a difficult to do as well. Sigh.

I still think it would be great to have python-poppler-qt5 in conda-forge, but for now I went back to the goal of producing wheels. I've done a short exploration today and it seems that actually, building Poppler from source is not as bad as I was fearing. By disabling lots of build options, you can get a version of Poppler with a reasonable number of external shared libs.