Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build fails with libbpf > 0.8.x, can only build kernel when using exact submodule commit #49

Open
Apteryks opened this issue May 8, 2024 · 16 comments

Comments

@Apteryks
Copy link

Apteryks commented May 8, 2024

The build fails like:

[...]
/tmp/guix-build-dwarves-1.26.drv-0/source/btf_loader.c: In function ‘create_new_enumeration64’:
/tmp/guix-build-dwarves-1.26.drv-0/source/btf_loader.c:350:54: error: invalid use of undefined type ‘struct btf_enum64’
  350 |                 const char *name = cu__btf_str(cu, ep[i].name_off);
      |                                                      ^
/tmp/guix-build-dwarves-1.26.drv-0/source/btf_loader.c:350:57: error: invalid use of undefined type ‘struct btf_enum64’
  350 |                 const char *name = cu__btf_str(cu, ep[i].name_off);
      |                                                         ^
/tmp/guix-build-dwarves-1.26.drv-0/source/btf_loader.c:351:54: error: invalid use of undefined type ‘struct btf_enum64’
  351 |                 uint64_t value = btf_enum64_value(&ep[i]);
      |                                                      ^
make[2]: *** [CMakeFiles/dwarves.dir/build.make:163: CMakeFiles/dwarves.dir/btf_loader.c.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/tmp/guix-build-dwarves-1.26.drv-0/build'
make[1]: *** [CMakeFiles/Makefile2:110: CMakeFiles/dwarves.dir/all] Error 2
make[1]: Leaving directory '/tmp/guix-build-dwarves-1.26.drv-0/build'
make: *** [Makefile:139: all] Error 2
@Apteryks
Copy link
Author

Apteryks commented May 8, 2024

I've reproduced the above using the latest available commit, d584761.

@Apteryks Apteryks changed the title Build fails with latest libbpf v1.4.1 Build fails with latest libbpf v1.4.1 (actually, any version > 1.1.0) May 8, 2024
@Apteryks Apteryks changed the title Build fails with latest libbpf v1.4.1 (actually, any version > 1.1.0) Build fails with latest libbpf v1.4.1 (actually, any version >= 1.0) May 8, 2024
@Apteryks
Copy link
Author

Apteryks commented May 8, 2024

I had to use libbpf 0.8.3.

@Apteryks
Copy link
Author

Apteryks commented May 8, 2024

Although it built fine, compiling the Linux kernel with CONFIG_DEBUG_INFO would fail with a failed to find '.BTF' ELF section in vmlinux error message; using the exact same commit as the libbpf submodule was needed.

It seems dwarves is very tightly coupled to the inner workings of libbpf? It'd be nice if it targetted at least a released version of libbpf.

@Apteryks Apteryks changed the title Build fails with latest libbpf v1.4.1 (actually, any version >= 1.0) Build fails with latest libbpf > 0.8.x, can only build kernel when using exact submodule commit May 8, 2024
@Apteryks Apteryks changed the title Build fails with latest libbpf > 0.8.x, can only build kernel when using exact submodule commit Build fails with libbpf > 0.8.x, can only build kernel when using exact submodule commit May 8, 2024
@martinetd
Copy link

nixpkgs builds pahole with libbpf v1.4.1, so either of us are doing something wrong - please provide a reproducer or explain how you're building

$ ldd /nix/store/j6yi4ngcwf423igmqspjdxvkw7g3mvgn-pahole-1.26/lib/libdwarves.so.1.0.0 | grep bpf
	libbpf.so.1 => /nix/store/bbj6lrp82h79viq74pp8fzwdbag8l80n-libbpf-1.4.1/lib/libbpf.so.1 (0x00007fc14afea000)

@Apteryks
Copy link
Author

Hi! It's fairly easy to reproduce for me, using libbpf v1.4.1, ensuring I don't pull the git submodules of dwarves, specifying the "-DLIBBPF_EMBEDDED=OFF" build option and providing pkg-config as a build input:

starting phase `configure'
source directory: "/tmp/guix-build-dwarves-1.26.drv-0/source" (relative from build: "../source")
build directory: "/tmp/guix-build-dwarves-1.26.drv-0/build"
running 'cmake' with arguments ("../source" "-DCMAKE_BUILD_TYPE=RelWithDebInfo" "-DCMAKE_INSTALL_PREFIX=/gnu/store/xgnss5xh7vhjwxl5mbmcpar4kzhfbdl8-dwarves-1.26" "-DCMAKE_INSTALL_LIBDIR=lib" "-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=TRUE" "-DCMAKE_INSTALL_RPATH=/gnu/store/xgnss5xh7vhjwxl5mbmcpar4kzhfbdl8-dwarves-1.26/lib" "-DCMAKE_VERBOSE_MAKEFILE=ON" "-D__LIB=lib" "-DLIBBPF_EMBEDDED=OFF")
-- The C compiler identification is GNU 11.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /gnu/store/5lqhcv91ijy82p92ac6g5xw48l0lwwz4-gcc-11.3.0/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found PkgConfig: /gnu/store/jz5dwdxq4di29cd0rjjzkw356dhkzjil-pkg-config-0.29.2/bin/pkg-config (found version "0.29.2") 
-- Checking for module 'libbpf>=0.4.0'
--   Found libbpf, version 1.4.1
-- Setting BUILD_SHARED_LIBS = ON
-- Checking availability of DWARF and ELF development libraries
-- Looking for dwfl_module_build_id in elf
-- Looking for dwfl_module_build_id in elf - found
-- Found dwarf.h header: /gnu/store/iyaad74kw54jrqzpwm5r4jagfr8dgirp-elfutils-0.187/include
-- Found elfutils/libdw.h header: /gnu/store/iyaad74kw54jrqzpwm5r4jagfr8dgirp-elfutils-0.187/include
-- Found libdw library: /gnu/store/iyaad74kw54jrqzpwm5r4jagfr8dgirp-elfutils-0.187/lib/libdw.so
-- Found libelf library: /gnu/store/iyaad74kw54jrqzpwm5r4jagfr8dgirp-elfutils-0.187/lib/libelf.so
-- Checking availability of DWARF and ELF development libraries - done
-- Found ZLIB: /gnu/store/slzq3zqwj75lbrg4ly51hfhbv2vhryv5-zlib-1.2.13/lib/libz.so (found version "1.2.13") 
-- Checking availability of argp library
-- Assuming argp is in libc
-- Checking availability of argp library - done
-- Checking availability of obstack library
-- Assuming obstack is in libc
-- Checking availability of obstack library - done
-- Performing Test HAVE_REALLOCARRAY_SUPPORT
-- Performing Test HAVE_REALLOCARRAY_SUPPORT - Success
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_INSTALL_LIBDIR


-- Build files have been written to: /tmp/guix-build-dwarves-1.26.drv-0/build
phase `configure' succeeded after 1.8 seconds
starting phase `patch-generated-file-shebangs'
phase `patch-generated-file-shebangs' succeeded after 0.0 seconds
starting phase `build'
/gnu/store/gl26kr5v6ch5lc3ignly61kb224drijc-cmake-minimal-3.24.2/bin/cmake -S/tmp/guix-build-dwarves-1.26.drv-0/source -B/tmp/guix-build-dwarves-1.26.drv-0/build --check-build-system CMakeFiles/Makefile.cmake 0
/gnu/store/gl26kr5v6ch5lc3ignly61kb224drijc-cmake-minimal-3.24.2/bin/cmake -E cmake_progress_start /tmp/guix-build-dwarves-1.26.drv-0/build/CMakeFiles /tmp/guix-build-dwarves-1.26.drv-0/build//CMakeFiles/progress.marks
make  -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/tmp/guix-build-dwarves-1.26.drv-0/build'
make  -f CMakeFiles/dwarves.dir/build.make CMakeFiles/dwarves.dir/depend
make[2]: Entering directory '/tmp/guix-build-dwarves-1.26.drv-0/build'
cd /tmp/guix-build-dwarves-1.26.drv-0/build && /gnu/store/gl26kr5v6ch5lc3ignly61kb224drijc-cmake-minimal-3.24.2/bin/cmake -E cmake_depends "Unix Makefiles" /tmp/guix-build-dwarves-1.26.drv-0/source /tmp/guix-build-dwarves-1.26.drv-0/source /tmp/guix-build-dwarves-1.26.drv-0/build /tmp/guix-build-dwarves-1.26.drv-0/build /tmp/guix-build-dwarves-1.26.drv-0/build/CMakeFiles/dwarves.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/tmp/guix-build-dwarves-1.26.drv-0/build'
make  -f CMakeFiles/dwarves.dir/build.make CMakeFiles/dwarves.dir/build
make[2]: Entering directory '/tmp/guix-build-dwarves-1.26.drv-0/build'
[  2%] Building C object CMakeFiles/dwarves.dir/dwarves.c.o
/gnu/store/5lqhcv91ijy82p92ac6g5xw48l0lwwz4-gcc-11.3.0/bin/gcc -DDWARVES_MAJOR_VERSION=1 -DDWARVES_MINOR_VERSION=26 -D_GNU_SOURCE -Ddwarves_EXPORTS -I/tmp/guix-build-dwarves-1.26.drv-0/build -I/tmp/guix-build-dwarves-1.26.drv-0/source -pthread -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DNDEBUG -fPIC -MD -MT CMakeFiles/dwarves.dir/dwarves.c.o -MF CMakeFiles/dwarves.dir/dwarves.c.o.d -o CMakeFiles/dwarves.dir/dwarves.c.o -c /tmp/guix-build-dwarves-1.26.drv-0/source/dwarves.c
[  5%] Building C object CMakeFiles/dwarves.dir/gobuffer.c.o
/gnu/store/5lqhcv91ijy82p92ac6g5xw48l0lwwz4-gcc-11.3.0/bin/gcc -DDWARVES_MAJOR_VERSION=1 -DDWARVES_MINOR_VERSION=26 -D_GNU_SOURCE -Ddwarves_EXPORTS -I/tmp/guix-build-dwarves-1.26.drv-0/build -I/tmp/guix-build-dwarves-1.26.drv-0/source -pthread -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DNDEBUG -fPIC -MD -MT CMakeFiles/dwarves.dir/gobuffer.c.o -MF CMakeFiles/dwarves.dir/gobuffer.c.o.d -o CMakeFiles/dwarves.dir/gobuffer.c.o -c /tmp/guix-build-dwarves-1.26.drv-0/source/gobuffer.c
[  8%] Building C object CMakeFiles/dwarves.dir/dwarves_fprintf.c.o
/gnu/store/5lqhcv91ijy82p92ac6g5xw48l0lwwz4-gcc-11.3.0/bin/gcc -DDWARVES_MAJOR_VERSION=1 -DDWARVES_MINOR_VERSION=26 -D_GNU_SOURCE -Ddwarves_EXPORTS -I/tmp/guix-build-dwarves-1.26.drv-0/build -I/tmp/guix-build-dwarves-1.26.drv-0/source -pthread -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DNDEBUG -fPIC -MD -MT CMakeFiles/dwarves.dir/dwarves_fprintf.c.o -MF CMakeFiles/dwarves.dir/dwarves_fprintf.c.o.d -o CMakeFiles/dwarves.dir/dwarves_fprintf.c.o -c /tmp/guix-build-dwarves-1.26.drv-0/source/dwarves_fprintf.c
[ 10%] Building C object CMakeFiles/dwarves.dir/ctf_loader.c.o
/gnu/store/5lqhcv91ijy82p92ac6g5xw48l0lwwz4-gcc-11.3.0/bin/gcc -DDWARVES_MAJOR_VERSION=1 -DDWARVES_MINOR_VERSION=26 -D_GNU_SOURCE -Ddwarves_EXPORTS -I/tmp/guix-build-dwarves-1.26.drv-0/build -I/tmp/guix-build-dwarves-1.26.drv-0/source -pthread -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DNDEBUG -fPIC -MD -MT CMakeFiles/dwarves.dir/ctf_loader.c.o -MF CMakeFiles/dwarves.dir/ctf_loader.c.o.d -o CMakeFiles/dwarves.dir/ctf_loader.c.o -c /tmp/guix-build-dwarves-1.26.drv-0/source/ctf_loader.c
[ 13%] Building C object CMakeFiles/dwarves.dir/libctf.c.o
/gnu/store/5lqhcv91ijy82p92ac6g5xw48l0lwwz4-gcc-11.3.0/bin/gcc -DDWARVES_MAJOR_VERSION=1 -DDWARVES_MINOR_VERSION=26 -D_GNU_SOURCE -Ddwarves_EXPORTS -I/tmp/guix-build-dwarves-1.26.drv-0/build -I/tmp/guix-build-dwarves-1.26.drv-0/source -pthread -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DNDEBUG -fPIC -MD -MT CMakeFiles/dwarves.dir/libctf.c.o -MF CMakeFiles/dwarves.dir/libctf.c.o.d -o CMakeFiles/dwarves.dir/libctf.c.o -c /tmp/guix-build-dwarves-1.26.drv-0/source/libctf.c
[ 16%] Building C object CMakeFiles/dwarves.dir/btf_encoder.c.o
/gnu/store/5lqhcv91ijy82p92ac6g5xw48l0lwwz4-gcc-11.3.0/bin/gcc -DDWARVES_MAJOR_VERSION=1 -DDWARVES_MINOR_VERSION=26 -D_GNU_SOURCE -Ddwarves_EXPORTS -I/tmp/guix-build-dwarves-1.26.drv-0/build -I/tmp/guix-build-dwarves-1.26.drv-0/source -pthread -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DNDEBUG -fPIC -MD -MT CMakeFiles/dwarves.dir/btf_encoder.c.o -MF CMakeFiles/dwarves.dir/btf_encoder.c.o.d -o CMakeFiles/dwarves.dir/btf_encoder.c.o -c /tmp/guix-build-dwarves-1.26.drv-0/source/btf_encoder.c
[ 18%] Building C object CMakeFiles/dwarves.dir/btf_loader.c.o
/gnu/store/5lqhcv91ijy82p92ac6g5xw48l0lwwz4-gcc-11.3.0/bin/gcc -DDWARVES_MAJOR_VERSION=1 -DDWARVES_MINOR_VERSION=26 -D_GNU_SOURCE -Ddwarves_EXPORTS -I/tmp/guix-build-dwarves-1.26.drv-0/build -I/tmp/guix-build-dwarves-1.26.drv-0/source -pthread -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DNDEBUG -fPIC -MD -MT CMakeFiles/dwarves.dir/btf_loader.c.o -MF CMakeFiles/dwarves.dir/btf_loader.c.o.d -o CMakeFiles/dwarves.dir/btf_loader.c.o -c /tmp/guix-build-dwarves-1.26.drv-0/source/btf_loader.c
/tmp/guix-build-dwarves-1.26.drv-0/source/btf_loader.c: In function create_new_enumeration64:
/tmp/guix-build-dwarves-1.26.drv-0/source/btf_loader.c:350:54: error: invalid use of undefined type struct btf_enum64
  350 |                 const char *name = cu__btf_str(cu, ep[i].name_off);
      |                                                      ^
/tmp/guix-build-dwarves-1.26.drv-0/source/btf_loader.c:350:57: error: invalid use of undefined type struct btf_enum64
  350 |                 const char *name = cu__btf_str(cu, ep[i].name_off);
      |                                                         ^
/tmp/guix-build-dwarves-1.26.drv-0/source/btf_loader.c:351:54: error: invalid use of undefined type struct btf_enum64
  351 |                 uint64_t value = btf_enum64_value(&ep[i]);
      |                                                      ^
make[2]: *** [CMakeFiles/dwarves.dir/build.make:163: CMakeFiles/dwarves.dir/btf_loader.c.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/tmp/guix-build-dwarves-1.26.drv-0/build'
make[1]: *** [CMakeFiles/Makefile2:110: CMakeFiles/dwarves.dir/all] Error 2
make[1]: Leaving directory '/tmp/guix-build-dwarves-1.26.drv-0/build'
make: *** [Makefile:139: all] Error 2
error: in phase 'build': uncaught exception:
%exception #<&invoke-error program: "make" arguments: ("-j" "4") exit-status: 2 term-signal: #f stop-signal: #f> 
phase `build' failed after 1.7 seconds
command "make" "-j" "4" failed with status 2
builder for `/gnu/store/b3rzaqikjqn2sf1m3m1lb1gqrc6kgybv-dwarves-1.26.drv' failed with exit code 1
build of /gnu/store/b3rzaqikjqn2sf1m3m1lb1gqrc6kgybv-dwarves-1.26.drv failed
View build log at '/var/log/guix/drvs/b3/rzaqikjqn2sf1m3m1lb1gqrc6kgybv-dwarves-1.26.drv'.
guix build: error: build of `/gnu/store/b3rzaqikjqn2sf1m3m1lb1gqrc6kgybv-dwarves-1.26.drv' failed

@Apteryks
Copy link
Author

I note that pahole on nixpkgs use a different git repo snapshot: https://git.kernel.org/pub/scm/devel/pahole/pahole.git/snapshot/pahole-1.26.tar.gz to build it, though it fails the same for me if I use guix build dwarves --with-source=dwarves=https://git.kernel.org/pub/scm/devel/pahole/pahole.git/snapshot/pahole-1.26.tar.gz.

@Apteryks
Copy link
Author

I found the underlying issue: libbpf/libbpf#562

@Apteryks
Copy link
Author

I note that the default headers used in the build environment on Guix are at 5.15.49, while Nix uses 6.7, which is probably why it works there.

@Apteryks
Copy link
Author

Apteryks commented May 10, 2024

If I add the headers of 6.8 to the build environment, it builds fine, so it seems the #562 fix of libbpf is insufficient, at least for kernel headers as old as 5.15.49.

@Apteryks
Copy link
Author

Fails as well on 5.15.157. I'll report that the fix doesn't work for 5.15.X on the libbpf issue.

@martinetd
Copy link

If I add the headers of 6.8 to the build environment, it build fine, so it seems the #562 fix of libbpf is insufficient, at least kernel headers as old as 5.15.49.

It's a bit worse than that, for bpf-related headers we install the headers that were shipped in libbpf, so we always build with whatever is compatible with libbpf (this is actually a bit of a hack as I don't see where we give it priority over linux headers... but it seems to work out in practice) without worrying about what kernel includes are used:

  postInstall = ''
    # install linux's libbpf-compatible linux/btf.h
    install -Dm444 include/uapi/linux/*.h -t $out/include/linux
  '';

(We do this because we also had quite a few problems with older linux headers in the past, as some libbpf releases depended on non-released kernels. using newer libbpf headers is fine as kernel headers are guaranteed to be backwards compatible.)

Either way it's not a dwarves problem, you probably cannot build anything with libbpf in that configuration (newer libbpf and older linux/bpf.h), so this probably should be closed in favor of a new issue in libbpf.

@Apteryks
Copy link
Author

Apteryks commented May 10, 2024

I've pinged the libbpf people at: libbpf/libbpf#562 (comment).

@martinetd
Copy link

Sorry, I should have checked the actual error here before speaking, it's not really a libbpf problem.

libbpf 1.x provides APIs that use btf_enum64, but the struct btf_enum64 definition itself comes from the linux uapi headers, so libbpf/libbpf#562 rightfully made libbpf's btf.h not depend on the struct definition (so code that doesn't use these functions can build with older kernel heades), but using the functions themselves require manipulating the structs so the pahole code needs the definition.

I guess pahole could add a check for it and either skip the code or redefine the enum, but there's no easy define to check for this so frankly you'll be better off upgrading your linux headers or using libbpf's uapi files like nix does...

@Apteryks
Copy link
Author

Hm, upgrading the linux-kernel-headers will have to wait until the next core updates iteration for Guix. Meanwhile copying the whole Linux 6.8 headers would add 6.8 MiB to the size of libbpf. Not great. Perhaps I could do with just bpf.h and types.h and a few others...

@Apteryks
Copy link
Author

Also, what is uapi ? I don't see this directory in the headers of our linux-libre-headers package.

@Apteryks
Copy link
Author

Apteryks commented May 10, 2024

I've added the following phase to libbpf, which emulates what's done in Nix using the least amount of Linux headers:

(add-after 'install 'install-linux-bpf-headers
            ;; Workaround users such as 'dwarves' requiring btf_enum64
            ;; definition from the kernel Linux >= 6 headers (see:
            ;; https://github.com/acmel/dwarves/issues/49).
            ;; TODO: Remove once our 'linux-libre-headers' package is
            ;; upgraded to a >= 6 release.
            (lambda _
              (let ((linux-libre-headers #$(this-package-native-input
                                            "linux-libre-headers")))
                (for-each (lambda (f)
                            (install-file (string-append linux-libre-headers
                                                         "/include/" f)
                                          (string-append #$output "/include/"
                                                         (dirname f))))
                          ;; This list contains btf.h and its transitive
                          ;; dependencies.
                          (list "asm/posix_types.h"
                                "asm/types.h"
                                "asm-generic/types.h"
                                "asm-generic/int-ll64.h"
                                "linux/btf.h"
                                "linux/posix_types.h"
                                "linux/stddef.h"
                                "linux/types.h")))))

It adds about 100 KiB of overhead, and appears to resolve the issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants