build: don't generate fake GNU libtool .la files

[orbea]

When building on unix systems TigerVNC will build several static archives (.a) and corresponding GNU libtool archive (.la) files within cmake build files. However these .la files are specific to GNU libtool and are not compatible with alternative libtool implementations such as Slibtool which results in build issues.

Instead it is better to link directly with the static archives which both GNU libtool and Slibtool can consume, but this requires the dependency_libs variable from the fake libtool archives which are now generated in automake (unix.mk) files and then included in Makefile.am.

Gentoo bug: https://bugs.gentoo.org/913581

Note that there is more clean up in the CMakeMacroLibtoolFile.cmake file that can be done, perhaps even changing its name, but as a build fix I tried to make this commit as small as possible.

[CendioOssman]

Thanks for your submission!

I'm a bit concerned about just dumping the whole .la approach, though. That is what Xorg's build system uses normally, after all.

Can't we try to fix the incompatibility in Slibtool instead?

Ideally in Slibtool, since I would assume it is a bug there since libtool can handle these files just fine?

But as a second choice, try to work around it in our end. I don't quite understand what Slibtool is upset about though from that bug report. Can you attempt to clarify what the problem is?

[orbea]

I can understand your concern, but given that TigerVNC mixes cmake and autotools that puts the build issue in a unique position. GNU libtool and slibtool are not compatible when used within the same build invocation which can be evidenced by the libvnc.la file generated with slibtool.

# libtool compatible library wrapper
# Generated by rlibtool (slibtool 0.5.34)
# [commit reference: a126a7f68ef374d65d014226195380eb7dceb634]

dlname='libvnc.so'
library_names='libvnc.so libvnc.so libvnc.so'
old_library='liblibvnc.a'

inherited_linker_flags=''
dependency_libs=''
weak_library_names=''

current=0
age=0
revision=0

installed=no
shouldnotlink=no

dlopen=''
dlpreopen=''

libdir='/usr/lib/xorg/modules/extensions'

And with GNU libtool.

# libvnc.la - a libtool library file
# Generated by libtool (GNU libtool) 2.4.7
#
# Please DO NOT delete this file!
# It is necessary for linking the library.

# The name that we can dlopen(3).
dlname='libvnc.so'

# Names of this library.
library_names='libvnc.so libvnc.so libvnc.so'

# The name of the static archive.
old_library=''

# Linker flags that cannot go in dependency_libs.
inherited_linker_flags=''

# Libraries that this one depends upon.
dependency_libs=' -L/usr/lib -ljpeg -lpixman-1 -lavcodec -lavutil -lswscale -lpam -lhogweed -lgmp -lz -lnettle -lbsd'

# Names of additional weak libraries provided by this library
weak_library_names=''

# Version information for libvnc.
current=0
age=0
revision=0

# Is this an already installed library?
installed=no

# Should we warn about portability when linking against -modules?
shouldnotlink=yes

# Files to dlopen/dlpreopen
dlopen=''
dlpreopen=''

# Directory that this library needs to be installed in:
libdir='/usr/lib/xorg/modules/extensions'

Additionally looking at the unix/xserver/hw/vnc/.libs it can be seen how they generate different files.

--- tigervnc-libtool.libs	2023-09-06 06:57:13.339030850 -0700
+++ tigervnc-slibtool.libs	2023-09-06 06:54:06.153949071 -0700
@@ -1,9 +1,21 @@
+.libs/Xvnc:                                ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, not stripped
+.libs/Xvnc.exe.wrapper:                    symbolic link to ../Xvnc
+.libs/libvnc.a.disabled:                   symbolic link to /dev/null
 .libs/libvnc.la:                           symbolic link to ../libvnc.la
-.libs/libvnc.lai:                          libtool library file, ASCII text
+.libs/libvnc.lai:                          symbolic link to ../libvnc.la
 .libs/libvnc.so:                           ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
+.libs/libvnc.so.def:                       symbolic link to /dev/null
+.libs/libvnc.so.def.host:                  symbolic link to libvnc.so.def.linux
+.libs/libvnc.so.def.linux:                 symbolic link to libvnc.so.def
+.libs/libvnc.so.slibtool.deps:             ASCII text
+.libs/libvnc.so.slibtool.rpath:            symbolic link to /usr/lib/xorg/modules/extensions
 .libs/libvnc_la-vncModule.o:               ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped
 .libs/libvnccommon.a:                      current ar archive
+.libs/libvnccommon.a.slibtool.deps:        ASCII text
 .libs/libvnccommon.la:                     symbolic link to ../libvnccommon.la
+.libs/libvnccommon.lai:                    symbolic link to ../libvnccommon.la
+.libs/libvnccommon.so.def.host:            broken symbolic link to libvnccommon.so.def.linux
+.libs/libvnccommon.so.def.linux:           broken symbolic link to libvnccommon.so.def
 .libs/libvnccommon_la-RFBGlue.o:           ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped
 .libs/libvnccommon_la-RandrGlue.o:         ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped
 .libs/libvnccommon_la-XorgGlue.o:          ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped

I am doubtful that slibtool is the right place to fix this nor do I think their upstream is going to want to do that.

This leaves a few choices that I can see:

• Find a more compatible way to use dependencies created by cmake within autotools such as this PR.
• Stop using cmake and use autotools so that these files are generated with the `$(LIBTOOL) implementation.
• Or the xserver git repo now uses meson and not autotools so this problem will entirely go away then, but the most current release (21.1.8) still has the autotools build system.

Given that future xserver releases will be different I think the first and third choice are the better ones.

[CendioOssman]

I looked a bit closer at this, and slibtool do state that they don't actually use any existing .la files:

slibtool's behavior is to always produce .la wrappers on the one hand, yet fully ignore their content on the other.

On the other hand, they also claim:

Despite their internal nature, installed .la wrappers are often [ab]used in strange and mysterious ways by distro-related tools other than libtool itself. For the sake of distributions that depend on the aforementioned wrappers, slibtool comes with three special symlinks named clibtool, clibtool-shared, and clibtool-static, respectively. The 'c' in clibtool stands for compatible, and accordingly indicates an end-user's preference towards perfect emulation of legacy behavior.

I tried using the "c" version, and it still didn't use the .la files. I'm uncertain if this is a bug in slibtool, or if "perfect emulation" doesn't necessarily cover this use case.

Either way, I still maintain the opinion that if slibtool is to be a libtool replacement, then it needs to be able to consume libtool compatible .la files. Otherwise, slibtool is just another competing build system. Which we would need to justify maintaining support for. And slibtool seems to be a very niche tool, so I don't really see why we can't just tell people that standard libtool is what is supported?

Related to that, the change suggested in this PR doesn't seem to be worth the risk. I can already see one bug, in that it doesn't seem to handle propagation of dependencies recursively, which is need in some cases. That can likely be fixed, but I don't see the big upside of having to take on that burden of duplicating what libtool currently does for us.

Exploring meson is probably more interesting, as it is likely that we'll need to support that build system eventually anyway.

[midipix]

I looked a bit closer at this, and slibtool do state that they don't actually use any existing .la files:

slibtool's behavior is to always produce .la wrappers on the one hand, yet fully ignore their content on the other.

On the other hand, they also claim:

Despite their internal nature, installed .la wrappers are often [ab]used in strange and mysterious ways by distro-related tools other than libtool itself. For the sake of distributions that depend on the aforementioned wrappers, slibtool comes with three special symlinks named clibtool, clibtool-shared, and clibtool-static, respectively. The 'c' in clibtool stands for compatible, and accordingly indicates an end-user's preference towards perfect emulation of legacy behavior.

I tried using the "c" version, and it still didn't use the .la files. I'm uncertain if this is a bug in slibtool, or if "perfect emulation" doesn't necessarily cover this use case.

The compatibility "c" version exists so that one could install compatible .la files to the target directory. The content of .la files, and likewise the name and format of temporary files that are created as part of the build process (in the .libs directory), are considered an implementation-specific detail that one should not rely upon.

As things currently stand, tigervnc generates the .la files under the assumption that there exists only a single libtool implementation, namely GNU's. In C programming, that would be the same as writing one's code against glibc's internals rather than against the POSIX specification. Then again, GNU libtool itself might decide at some point to change its implementation internals (format of .la files, .libs directory structure, etc.), which would then result in tigervnc failing to build. (admittedly, this last scenario is highly unlikely 🤪)

As far as I can tell, the best and least intrusive solution would be to:

• have CMake continue to generate the individual .o (and .lo) files.
• shift the task of generating the static or dynamic libraries (and thereby their accompanying .la files) from CMake to $(LIBTOOL).