Investigate build in aarch64/arm64 on Apple Silicon #1

marcindulak · 2022-10-09T18:11:11Z

The project does not produce (yet?) an image bootable with qemu-system-aarch64 on Apple Silicon. Maybe the errors and notes provided here can be useful later.

Some observations:

The first indication that something is off is in cat $LFS/jhalfs/logs/505-gcc-libstdc++-12.2.0 of 505-gcc-libstdc++

rm: cannot remove '/vagrant/jhalfs/mnt/build_dir/usr/lib/libstdc++.la': No such file or directory
rm: cannot remove '/vagrant/jhalfs/mnt/build_dir/usr/lib/libstdc++fs.la': No such file or directory
rm: cannot remove '/vagrant/jhalfs/mnt/build_dir/usr/lib/libsupc++.la': No such file or directory

It can be notes that the la files are being written into the usr/lib4 folder instead.
For now the error can be ignored by modifying $LFS/jhalfs/lfs-commands/chapter05/505-gcc-libstdc++

 rm -vf $LFS/usr/lib/lib{stdc++,stdc++fs,supc++}.la

At the end of the lfs build, the following files may be present

 ls ./usr/lib64/*.la
 ./usr/lib64/libasan.la    ./usr/lib64/libgomp.la    ./usr/lib64/libssp.la            ./usr/lib64/libsupc++.la
 ./usr/lib64/libatomic.la  ./usr/lib64/libhwasan.la  ./usr/lib64/libssp_nonshared.la  ./usr/lib64/libtsan.la
 ./usr/lib64/libcc1.la     ./usr/lib64/libitm.la     ./usr/lib64/libstdc++.la         ./usr/lib64/libubsan.la
./usr/lib64/libffi.la     ./usr/lib64/liblsan.la    ./usr/lib64/libstdc++fs.la

The next issue appears for 803-glibc-2.36, where it hangs at make -k check.

 USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
 root     15626  0.0  0.0   6912  3072 pts/2    Ss+  21:14   0:00 su - vagrant -c source /vagrant/jhalfs/jhalfs.sh && cd $LFS/jhalfs && make mk_CHROOT
 vagrant  15632  0.0  0.0   2700  2060 ?        Ss   21:14   0:00  \_ make mk_CHROOT
 vagrant  19070  0.0  0.0   3872  1292 ?        S    21:50   0:00      \_ /bin/bash -c ( sudo /usr/sbin/chroot /vagrant/jhalfs/mnt/build_dir /usr/bin/env -i HOME=/root TERM="$TERM" PS1='(lfs chroot) \u:\w\$ ' PATH=/usr/bin:/usr/sbin /bin/bash --login -c "cd jhalfs && make BREAKPOINT= CHROOT")
 root     19071  0.0  0.0   7820  3912 ?        S    21:50   0:00          \_ sudo /usr/sbin/chroot /vagrant/jhalfs/mnt/build_dir /usr/bin/env -i HOME=/root TERM=xterm PS1=(lfs chroot) \u:\w\$  PATH=/usr/bin:/usr/sbin /bin/bash --login -c cd jhalfs && make BREAKPOINT= CHROOT
 root     19072  0.0  0.0   2620  1844 ?        S    21:50   0:00              \_ make BREAKPOINT= CHROOT
 root      2118  0.0  0.0   3960  2388 ?        S    21:58   0:00                  \_ /bin/bash -c source envars && \ /jhalfs/lfs-commands/chapter08/803-glibc >>/jhalfs/logs/803-glibc-2.36 2>&1
 root      2119  0.0  0.0   3960  2808 ?        S    21:58   0:00                      \_ /bin/bash /jhalfs/lfs-commands/chapter08/803-glibc
 root     11111  0.0  0.0   2516  1764 ?        S    22:01   0:00                          \_ make -k check
 root     15444  0.0  0.0   4136  3440 pts/1    Ss   21:11   0:00 bash
 root     24377  0.0  0.0   6408  1640 pts/1    R+   22:25   0:00  \_ ps faux
 root         1  0.0  0.0   2308   752 pts/0    Ss+  21:08   0:00 sh -c  sleep infinity
 root         7  0.0  0.0   2204   764 pts/0    S+   21:08   0:00 sleep infinity
 root      2097  0.0  0.0   3960  1948 ?        S    21:58   0:00 /bin/bash -c export BASHBIN=/bin/bash && /bin/bash progress_bar.sh 803-glibc $PPID &
 root      2099  0.3  0.0   3960  2816 ?        S    21:58   0:04  \_ /bin/bash progress_bar.sh 803-glibc 19072
 root     24376  0.0  0.0   2156   236 ?        S    22:25   0:00      \_ /bin/sleep .12
 root      5040  0.0  0.0   2324   220 ?        S    22:07   0:00 /sources/glibc-2.36/build/elf/ld-linux-aarch64.so.1 --library-path /sources/glibc-2.36/build:/sources/glibc-2.36/build/math:/sources/glibc-2.36/build/elf:/sources/glibc-2.36/build/dlfcn:/sources/glibc-2.36/build/nss:/sources/glibc-2.36/build/nis:/sources/glibc-2.36/build/rt:/sources/glibc-2.36/build/resolv:/sources/glibc-2.36/build/mathvec:/sources/glibc-2.36/build/support:/sources/glibc-2.36/build/crypt:/sources/glibc-2.36/build/nptl /sources/glibc-2.36/build/nptl/tst-cancel7 --direct --sem /tmp/tst-cancel7.l14yVI --pidfile /tmp/tst-cancel7-pid-juVafC
 root     29992  0.0  0.0   2324   224 ?        S    22:09   0:00 /sources/glibc-2.36/build/elf/ld-linux-aarch64.so.1 --library-path /sources/glibc-2.36/build:/sources/glibc-2.36/build/math:/sources/glibc-2.36/build/elf:/sources/glibc-2.36/build/dlfcn:/sources/glibc-2.36/build/nss:/sources/glibc-2.36/build/nis:/sources/glibc-2.36/build/rt:/sources/glibc-2.36/build/resolv:/sources/glibc-2.36/build/mathvec:/sources/glibc-2.36/build/support:/sources/glibc-2.36/build/crypt:/sources/glibc-2.36/build/nptl /sources/glibc-2.36/build/nptl/tst-cancelx7 --direct --sem /tmp/tst-cancel7.k5qiF4 --pidfile /tmp/tst-cancel7-pid-AILVrb

When kill -15 11111 is used, the target proceeds and finishes the without errors.
The logs cat $LFS/jhalfs/test-logs/803-glibc-2.36 shows the tests finished

Summary of test results:
    169 FAIL
   4440 PASS
     51 UNSUPPORTED
     16 XFAIL
      2 XPASS
make[2]: *** [Makefile:647: tests] Error 1
make[2]: Target 'check' not remade because of errors.
make[2]: Leaving directory '/sources/glibc-2.36'
make[1]: *** [Makefile:9: check] Error 2

and differ slightly vs the corresponding log on x86_64/amd64 (which does not require kill)

Summary of test results:
      5 FAIL
   5032 PASS
    124 UNSUPPORTED
     16 XFAIL
      4 XPASS
make[2]: *** [Makefile:647: tests] Error 1
make[2]: Target 'check' not remade because of errors.
make[2]: Leaving directory '/sources/glibc-2.36'
make[1]: *** [Makefile:9: check] Error 2
make[1]: Leaving directory '/sources/glibc-2.36/build'

The next error appears in 814-expect. The build log $LFS/jhalfs/logs/814-expect-5.45.4 shows the build platform $(uname -m) == aarch64) is uknown to expect

...
checking if running Sequent running SVR4... no
checking build system type... tclconfig/config.guess: unable to guess system type

This script, last modified 2003-10-07, has failed to recognize
the operating system you are using. It is advised that you
download the most up to date version of the config scripts from

 ftp://ftp.gnu.org/pub/gnu/config/
 
If the version you run (tclconfig/config.guess) is already up to date, please
send the following data and any information you think might be
pertinent to <[email protected]> in order to provide the needed
information to handle your system.

config.guess timestamp = 2003-10-07

uname -m = aarch64
uname -r = 5.15.64-0-virt
uname -s = Linux
uname -v = #1-Alpine SMP Mon, 05 Sep 2022 08:02:49 +0000

/usr/bin/uname -p = unknown
/bin/uname -X     =

hostinfo               =
/bin/universe          =
/usr/bin/arch -k       =
/bin/arch              =
/usr/bin/oslevel       =
/usr/convex/getsysinfo =

UNAME_MACHINE = aarch64
UNAME_RELEASE = 5.15.64-0-virt
UNAME_SYSTEM  = Linux
UNAME_VERSION = #1-Alpine SMP Mon, 05 Sep 2022 08:02:49 +0000
configure: error: cannot guess build type; you must specify one

This can be remediated by adjusting $LFS/jhalfs/fs-commands/chapter08/814-expect to use arm (not arm64)

 ./configure
         ...
         --build=arm
         ...

The next error happens in 824-gcc, the log cat jhalfs/logs/824-gcc-12.2.0 shows the build fails to find py files under ./user/lib

...
'/usr/lib/bfd-plugins/liblto_plugin.so' -> '../../libexec/gcc/aarch64-unknown-linux-gnu/12.2.0/liblto_plugin.so'
mkdir: created directory '/usr/share/gdb'
mkdir: created directory '/usr/share/gdb/auto-load'
mkdir: created directory '/usr/share/gdb/auto-load/usr'
mkdir: created directory '/usr/share/gdb/auto-load/usr/lib'
mv: cannot stat '/usr/lib/*gdb.py': No such file or directory

Let's ignore the error by commenting out the line mv -v /usr/lib/*gdb.py /usr/share/gdb/auto-load/usr/lib in $LFS/jhalfs/lfs-commands/chapter08/824-gcc and proceed.

The next issue appears in 847-libffi, but can only be noticed when trying to build 851-meson, whose log cat $LFS/jhalfs/logs/851-meson-0.63.2 shows

Processing /sources/meson-0.63.2
 Preparing metadata (pyproject.toml): started
 Preparing metadata (pyproject.toml): finished with status 'error'
 error: subprocess-exited-with-error

 × Preparing metadata (pyproject.toml) did not run successfully.
 │ exit code: 1
 ╰─> [64 lines of output]
     /usr/lib/python3.10/site-packages/setuptools/config/setupcfg.py:463: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
       warnings.warn(msg, warning_class)
     running dist_info
     creating /tmp/pip-modern-metadata-_hbvr78y/meson.egg-info
     ...
       File "/usr/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 26, in <module>
         from .macosx_libfile import calculate_macosx_platform_tag
       File "/usr/lib/python3.10/site-packages/wheel/macosx_libfile.py", line 41, in <module>
         import ctypes
       File "/usr/lib/python3.10/ctypes/__init__.py", line 8, in <module>
         from _ctypes import Union, Structure, Array
     ModuleNotFoundError: No module named '_ctypes'
     [end of output]

 note: This error originates from a subprocess, and is likely not a problem with pip.
 error: metadata-generation-failed
....

As a workaroun libffi can to configured with --disable-multi-os-directory flag in $LFS/jhalfs/lfs-commands/chapter08/847-libffi, rebuilt, then python also rebuilt. Meson should then install without any modifications.

The next build issue appears for 856-findutils whose build log cat $LFS/jhalfs/logs/856-findutils-4.9.0 reports there is no Makefile
```
make[1]: Entering directory '/sources/findutils-4.9.0'
There seems to be no Makefile in this directory.
You must run ./configure before running 'make'.
make[1]: *** [GNUmakefile:108: abort-due-to-no-makefile] Error 1
make[1]: Leaving directory '/sources/findutils-4.9.0'
```
This can be remediated in $LFS/jhalfs/lfs-commands/chapter08/856-findutils which needs the aarch64 platform added x86_64|aarch64) ./configure --prefix=/usr --localstatedir=/var/lib/locate ;;.

At the time the error is found in 873-procps-ng-4.0.0 log cat $LFS/jhalfs/logs/873-procps-ng-4.0.0 it becomes clear that many libraries are placed in ./usr/lib64 and this location is not used for configure

checking for libsystemd-login... no
configure: error: Package requirements (libsystemd-login) were not met:

No package 'libsystemd-login' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables SYSTEMD_CFLAGS
and SYSTEMD_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.

As a drastic workaround, linking everthing present under ./usr/lib64 can be considered

cd $LFS/usr/lib
for file in $(find ../lib64 -maxdepth 1 -type f); do ln -sv $file .; done
'./libtsan.a' -> '../lib64/libtsan.a'
'./libhwasan.so.0.0.0' -> '../lib64/libhwasan.so.0.0.0'
'./libstdc++.la' -> '../lib64/libstdc++.la'
'./libasan.a' -> '../lib64/libasan.a'
'./libasan.so.8.0.0' -> '../lib64/libasan.so.8.0.0'
'./libgcc_s.so' -> '../lib64/libgcc_s.so'
'./libsanitizer.spec' -> '../lib64/libsanitizer.spec'
'./libgomp.a' -> '../lib64/libgomp.a'
'./libasan_preinit.o' -> '../lib64/libasan_preinit.o'
'./libstdc++.so.6.0.30' -> '../lib64/libstdc++.so.6.0.30'
'./libssp_nonshared.la' -> '../lib64/libssp_nonshared.la'
'./libnss_resolve.so.2' -> '../lib64/libnss_resolve.so.2'
ln: failed to create symbolic link './libffi.so.8.1.1': File exists
'./liblsan.so.0.0.0' -> '../lib64/liblsan.so.0.0.0'
'./libsystemd.so.0.34.0' -> '../lib64/libsystemd.so.0.34.0'
'./libatomic.so.1.2.0' -> '../lib64/libatomic.so.1.2.0'
'./libstdc++fs.a' -> '../lib64/libstdc++fs.a'
'./libtsan.la' -> '../lib64/libtsan.la'
'./libgcc_s.so.1' -> '../lib64/libgcc_s.so.1'
'./liblsan.la' -> '../lib64/liblsan.la'
'./libstdc++.a' -> '../lib64/libstdc++.a'
'./libubsan.so.1.0.0' -> '../lib64/libubsan.so.1.0.0'
'./libhwasan.a' -> '../lib64/libhwasan.a'
'./libtsan_preinit.o' -> '../lib64/libtsan_preinit.o'
'./libcc1.la' -> '../lib64/libcc1.la'
'./libitm.spec' -> '../lib64/libitm.spec'
'./libatomic.a' -> '../lib64/libatomic.a'
'./libtsan.so.2.0.0' -> '../lib64/libtsan.so.2.0.0'
'./libnss_myhostname.so.2' -> '../lib64/libnss_myhostname.so.2'
'./libssp.la' -> '../lib64/libssp.la'
'./libatomic.la' -> '../lib64/libatomic.la'
'./libitm.so.1.0.0' -> '../lib64/libitm.so.1.0.0'
'./libubsan.a' -> '../lib64/libubsan.a'
'./libsupc++.a' -> '../lib64/libsupc++.a'
'./libnss_mymachines.so.2' -> '../lib64/libnss_mymachines.so.2'
'./libssp.so.0.0.0' -> '../lib64/libssp.so.0.0.0'
'./libstdc++fs.la' -> '../lib64/libstdc++fs.la'
'./libubsan.la' -> '../lib64/libubsan.la'
'./libssp_nonshared.a' -> '../lib64/libssp_nonshared.a'
'./libhwasan.la' -> '../lib64/libhwasan.la'
'./libasan.la' -> '../lib64/libasan.la'
'./libnss_systemd.so.2' -> '../lib64/libnss_systemd.so.2'
'./libitm.la' -> '../lib64/libitm.la'
ln: failed to create symbolic link './libffi.la': File exists
'./libsupc++.la' -> '../lib64/libsupc++.la'
'./libssp.a' -> '../lib64/libssp.a'
'./libstdc++.so.6.0.30-gdb.py' -> '../lib64/libstdc++.so.6.0.30-gdb.py'
'./liblsan.a' -> '../lib64/liblsan.a'
'./libitm.a' -> '../lib64/libitm.a'
'./libcc1.so.0.0.0' -> '../lib64/libcc1.so.0.0.0'
'./libudev.so.1.7.4' -> '../lib64/libudev.so.1.7.4'
'./libgomp.la' -> '../lib64/libgomp.la'
'./liblsan_preinit.o' -> '../lib64/liblsan_preinit.o'
'./libgomp.so.1.0.0' -> '../lib64/libgomp.so.1.0.0'
'./libgomp.spec' -> '../lib64/libgomp.spec'

cd pkgconfig
for file in $(find ../../lib64/pkgconfig -maxdepth 1 -type f); do ln -sv $file .; done
'./libsystemd.pc' -> '../../lib64/pkgconfig/libsystemd.pc'
'./libudev.pc' -> '../../lib64/pkgconfig/libudev.pc'

The build of 1002-kernel requires changing $LFS/jhalfs/lfs-commands/chapter10/1002-kernel to use Image instead of bzImage cp -v arch/arm64/boot/Image /boot/vmlinuz-5.19.13-lfs-r11.2-125-systemd+.
The boot process is unclear yet. The main question may be how to boot a BIOS based qemu image as compared to UEFI one as asked here and here. Currently qemu-system-aarch64 -m 256M -smp 1 -drive file=build_dir.img,format=raw,if=virtio -machine type=virt,accel=hvf,highmem=off -cpu cortex-a72 results only in qemu monitor shown. The surprising fact (?? verify this once more) is that even booting UEFI ubuntu arm64 as descibed in https://gist.github.com/nrjdalal/e70249bb5d2e9d844cc203fd11f74c55 stays at qemu monitor.

The text was updated successfully, but these errors were encountered:

pierre-labastie · 2022-10-10T06:34:44Z

It looks like you are using an x86 book for building on aarch64: this cannot work, although there are not so many changes needed. You can find an aarch64 lfs book at two places:

https://github.com/cross-lfs/lfs-arm
in the xry111/arm64 branch of https://git.linuxfromscratch.org/lfs.git
For the first one, you need to clone the repo, and use it as a working copy for jhalfs. For the second one, you can enter xry111/arm64 as the branch. Then all the build problem should be solved (well, I'm not the maintainer of those branches, so...).

For booting, I cannot help...

marcindulak · 2022-10-13T16:49:25Z

Thanks, I added notes from an attempted (and failed) aarch64 to https://github.com/cross-lfs/lfs-arm/issues/7. Hopefully I'll get some feedback there.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Investigate build in aarch64/arm64 on Apple Silicon #1

Investigate build in aarch64/arm64 on Apple Silicon #1

marcindulak commented Oct 9, 2022 •

edited

Loading

pierre-labastie commented Oct 10, 2022 •

edited

Loading

marcindulak commented Oct 13, 2022

Investigate build in aarch64/arm64 on Apple Silicon #1

Investigate build in aarch64/arm64 on Apple Silicon #1

Comments

marcindulak commented Oct 9, 2022 • edited Loading

pierre-labastie commented Oct 10, 2022 • edited Loading

marcindulak commented Oct 13, 2022

marcindulak commented Oct 9, 2022 •

edited

Loading

pierre-labastie commented Oct 10, 2022 •

edited

Loading