Build fails on aarch64 hosts

[msgilligan]

Update: The x86_64 build issue turned out to be non-reproducible, so this issue has been renamed. The actual problem I have been troubleshooting is building on aarch64 hosts and there are at least 3 sub-issues that have been discovered. This issue now serves as a parent issue for aarch64 host build issues.

On Debian 12 (x86_64), I'm following the standard build instructions for QEMU v8, e.g.

$ mkdir optee
$ cd optee
$ repo init -u https://github.com/OP-TEE/manifest.git -m qemu_v8.xml -b 4.2.0
$ repo sync
$ cd build
$ make toolchains
$ make run

The error I'm getting is:

  CC      lib/display_options.o
lib/display_options.c: In function ‘print_freq’:
lib/display_options.c:59:9: internal compiler error: Illegal instruction
   59 |         unsigned long d = 1e9;
      |         ^~~~~~~~
0x1739600 diagnostic_impl(rich_location*, diagnostic_metadata const*, int, char const*, __va_list_tag (*) [1], diagnostic_t)
	???:0
0x173a286 internal_error(char const*, ...)
	???:0
0xc26e2f crash_signal(int)
	???:0
0x193cec9 __gmpn_mul_basecase
	???:0
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <https://bugs.linaro.org/> for instructions.
make[2]: *** [scripts/Makefile.build:257: lib/display_options.o] Error 1
make[1]: *** [Makefile:1853: lib] Error 2
make[1]: Leaving directory '/home/sean/optee/u-boot'
make: *** [Makefile:305: u-boot] Error 2

[msgilligan]

Update: I tried the build with Ubuntu 22.04.4 LTS (Jammy Jellyfish) and am getting the same error. (I changed the title of the issue to reflect this.)

[jforissier]

Hi @msgilligan,

The build works for me with Ubuntu 22.04 (22.04.4 LTS) using the Dockerfile at https://optee.readthedocs.io/en/latest/building/prerequisites.html. It looks like for some reason you build is not picking up the proper cross-compiler. Could you try building with make V=1 and check the CC command used to build lib/display_options.o?

[msgilligan]

Hi @msgilligan,

The build works for me with Ubuntu 22.04 (22.04.4 LTS) using the Dockerfile at https://optee.readthedocs.io/en/latest/building/prerequisites.html. It looks like for some reason you build is not picking up the proper cross-compiler. Could you try building with make V=1 and check the CC command used to build lib/display_options.o?

Yes, when I run the same Dockerfile it works. I'm not sure what the difference is between Ubuntu running in a QEMU VM (using Lima on macOS) and the one in Docker.

I will try your suggestion and report back.

[msgilligan]

Here is the result:

make -f ./scripts/Makefile.build obj=lib/zlib
  /usr/bin/ccache /home/sean.linux/optee-qemu8/build/../toolchains/aarch64/bin/aarch64-linux-gnu-gcc -Wp,-MD,lib/.display_options.o.d -nostdinc -isystem /home/sean.linux/optee-qemu8/toolchains/aarch64/bin/../lib/gcc/aarch64-none-linux-gnu/11.3.1/include -Iinclude   -I./arch/arm/include -include ./include/linux/kconfig.h -D__KERNEL__ -D__UBOOT__ -Wall -Wstrict-prototypes -Wno-format-security -fno-builtin -ffreestanding -std=gnu11 -fshort-wchar -fno-strict-aliasing -fno-PIE -Os -fno-stack-protector -fno-delete-null-pointer-checks -Wno-pointer-sign -Wno-stringop-truncation -Wno-zero-length-bounds -Wno-array-bounds -Wno-stringop-overflow -Wno-maybe-uninitialized -fmacro-prefix-map=./= -gdwarf-4 -fstack-usage -Wno-format-nonliteral -Wno-address-of-packed-member -Wno-unused-but-set-variable -Werror=date-time -Wno-packed-not-aligned -D__ARM__ -fno-pic -mstrict-align -ffunction-sections -fdata-sections -fno-common -ffixed-x18 -mgeneral-regs-only -mbranch-protection=none -pipe -march=armv8-a+crc -D__LINUX_ARM_ARCH__=8    -DKBUILD_BASENAME='"display_options"'  -DKBUILD_MODNAME='"display_options"' -c -o lib/display_options.o lib/display_options.c
lib/display_options.c: In function ‘print_freq’:
lib/display_options.c:59:9: internal compiler error: Illegal instruction
   59 |         unsigned long d = 1e9;
      |         ^~~~~~~~
0x1739600 diagnostic_impl(rich_location*, diagnostic_metadata const*, int, char const*, __va_list_tag (*) [1], diagnostic_t)
	???:0
0x173a286 internal_error(char const*, ...)
	???:0
0xc26e2f crash_signal(int)
	???:0
0x193cec9 __gmpn_mul_basecase
	???:0
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <https://bugs.linaro.org/> for instructions.
make[2]: *** [scripts/Makefile.build:257: lib/display_options.o] Error 1
make[1]: *** [Makefile:1853: lib] Error 2
make[1]: Leaving directory '/home/sean.linux/optee-qemu8/u-boot'
make: *** [Makefile:305: u-boot] Error 2

[msgilligan]

I was able to successfully build the project in another Debian 12 environment and I compared the CC commands and they are the same. I'm working to narrow down other potential differences in the environment.

Of course, the code in display_options.c should probably be changed to unsigned long d = 1000000000; or maybe use an explicit cast.

[msgilligan]

@jforissier My goal is to be able to develop for OP-TEE on my M1 MacBook Pro. I have Docker installed and use Lima for long-lived development VMs. I prefer to use Debian (Bookworm) over Ubuntu, but am willing to use Ubuntu if necessary. But I do not want to use amd64/x86_64 emulation (builds take many hours), I want to run native arm64 VMs. I am willing to troubleshoot, document, and submit PRs to help make this happen. I can also move this to the mailing list, if that is a better venue.

I have verified that the reference Dockerfile works under Docker on my Mac when I run it with --platform linux/amd64, but without that setting under arm64 mode it fails. If I run the arm64 container with -j1 V=1 I get the following error message:

# Ensure the toolchain, components, and targets we've specified in
# rust-toolchain.toml are ready to go. Since that file sets rustup's
# default toolchain for the entire directory, all we need to do is run
# any rustup-wrapped command to trigger installation. We've arbitrarily
# chosen "cargo --version" since it has no other effect.
Configuring OP-TEE rust examples
/bin/bash: line 1: /.cargo/env: No such file or directory
make[2]: *** [package/pkg-generic.mk:273: /optee/out-br/build/optee_rust_examples_ext-1.0/.stamp_configured] Error 1
make[1]: *** [Makefile:23: _all] Error 2
make[1]: Leaving directory '/optee/out-br'
make: *** [common.mk:341: buildroot] Error 2
The command '/bin/sh -c make -j1 V=1' returned a non-zero code: 2

Although this is different from the symptom I initially reported, it seems like the first place to start, because in the case of a Docker build the environment is more carefully controlled and I'm seeing a difference in behavior based (apparently) solely on the CPU architecture.

I've also seen this problem (or similar) in builds inside VMs and on my Intel box. Under Docker the build seems to occur in the root directory, so /.cargo/env is a valid path (on amd64, at least).

From my perspective, It seems like there are at least two issues/todos to work on:

1. Identify and fix the reason for the arm64 Docker build failure.
2. Provide and/or (better) document a mechanism to configure the Rust toolchain path. (I've noticed a variable named BR2_PACKAGE_OPTEE_RUST_EXAMPLES_EXT_TC_PATH that can be used, but it's not documented on the website (e.g. on https://optee.readthedocs.io/en/latest/building/optee_with_rust.html)

On a separate (and positive!) note, when I do run the Docker build using amd64 and use Docker to copy /optee/out/bin/* out to a macOS directory, I can boot the Linux guest using a macOS build of QEMU and run xtest successfully.

I should also mention that when I did preliminary investigations in October 2023, I was able to do the full build in an arm64 VM on macOS and run xtest in a QEMU guest under macOS.

[msgilligan]

p.s. Using RUN make V=1 OPTEE_RUST_ENABLE=n does not seem to disable the use of Rust.

[jforissier]

Hi @msgilligan,

Being able to build natively on arm64 is a good thing obviously, and as you said it used to work. It should not be that hard to fix the issues and I appreciate your willingness to help in that regard.
Rust support has been enabled by default recently but was never tested on arm4 I believe :-/

p.s. Using RUN make V=1 OPTEE_RUST_ENABLE=n does not seem to disable the use of Rust.

That should be RUST_ENABLE=n, see https://github.com/OP-TEE/build/blob/4.2.0/qemu_v8.mk#L32-L33.

I've noticed a variable named BR2_PACKAGE_OPTEE_RUST_EXAMPLES_EXT_TC_PATH

Yes, it was introduced by f0a2eef. Basically it takes its value from RUST_TOOLCHAIN_PATH in common.mk RUST_TOOLCHAIN_PATH is set in toolchain.mk. make toolchains is supposed to install what's needed.

build$ git grep RUST_TOOLCHAIN_PATH
common.mk:BR2_PACKAGE_OPTEE_RUST_EXAMPLES_EXT_TC_PATH ?= $(RUST_TOOLCHAIN_PATH)
toolchain.mk:RUST_TOOLCHAIN_PATH                ?= $(TOOLCHAIN_ROOT)/rust
toolchain.mk:   $(call dl-rust-toolchain,$(RUST_TOOLCHAIN_PATH))

Does make toolchains work on your platform?

[msgilligan]

OPTEE_RUST_ENABLE

That should be RUST_ENABLE=n

This documentation page shows OPTEE_RUST_ENABLE :
https://github.com/OP-TEE/optee_docs/blob/9841a9fc14ed49b084bed4b6fd8af417a45cbf24/building/optee_with_rust.rst#L41
That's why I was trying that name.

Does make toolchains work on your platform?

Yes.

If I set RUST_ENABLE=n and run the Docker build on arm64, I get:

/usr/bin/ccache /optee/build/../toolchains/aarch64/bin/aarch64-linux-ld.bfd -e__ta_entry -pie -T out/ta/os_test/ta.lds -Map=out/ta/os_test/5b9e0e40-2636-11e1-ad9e-0002a5d5c51b.map --sort-section=alignment -z max-page-size=4096  --as-needed   --dynamic-list out/ta/os_test/dyn_list --eh-frame-hdr out/ta/os_test/init.o out/ta/os_test/os_test.o out/ta/os_test/ta_entry.o out/ta/os_test/test_float_subj.o out/ta/os_test/cxx_tests.o out/ta/os_test/cxx_tests_c.o out/ta/os_test/attestation.o out/ta/os_test/user_ta_header.o -L/optee/out-br/build/optee_test_ext-1.0/ta/os_test_lib/out/ta/os_test_lib -los_test -ldl -L/optee/optee_os/out/arm/export-ta_arm64/lib --start-group -lutils -lutee -lmbedtls -ldl /optee/toolchains/aarch64/bin/../lib/gcc/aarch64-buildroot-linux-gnu/12.3.0/../../../../aarch64-buildroot-linux-gnu/lib/../lib64/libstdc++.a /optee/toolchains/aarch64/bin/../lib/gcc/aarch64-buildroot-linux-gnu/12.3.0/libgcc_eh.a --end-group /optee/toolchains/aarch64/bin/../lib/gcc/aarch64-buildroot-linux-gnu/12.3.0/libgcc.a -lutils -o out/ta/os_test/5b9e0e40-2636-11e1-ad9e-0002a5d5c51b.elf
/optee/build/../toolchains/aarch64/bin/aarch64-linux-ld.bfd: /optee/toolchains/aarch64/bin/../lib/gcc/aarch64-buildroot-linux-gnu/12.3.0/libgcc_eh.a(unwind-dw2-fde-dip.o): in function `_Unwind_Find_FDE':
/optee/out-aarch64-sdk/build/host-gcc-final-12.3.0/build/aarch64-buildroot-linux-gnu/libgcc/../../../libgcc/unwind-dw2-fde-dip.c:512: undefined reference to `_dl_find_object'
make[4]: *** [/optee/optee_os/out/arm/export-ta_arm64/mk/link.mk:123: out/ta/os_test/5b9e0e40-2636-11e1-ad9e-0002a5d5c51b.elf] Error 1
make[3]: *** [/optee/out-br/build/optee_test_ext-1.0/ta/Makefile.gmake:61: ta-os_test] Error 2
make[2]: *** [package/pkg-generic.mk:284: /optee/out-br/build/optee_test_ext-1.0/.stamp_built] Error 2
make[1]: *** [Makefile:23: _all] Error 2
make[1]: Leaving directory '/optee/out-br'
make: *** [common.mk:341: buildroot] Error 2
The command '/bin/sh -c make V=1 RUST_ENABLE=n' returned a non-zero code: 2

[jforissier]

OPTEE_RUST_ENABLE

That should be RUST_ENABLE=n

This documentation page shows OPTEE_RUST_ENABLE : https://github.com/OP-TEE/optee_docs/blob/9841a9fc14ed49b084bed4b6fd8af417a45cbf24/building/optee_with_rust.rst#L41 That's why I was trying that name.

Ah, yes. The doc needs updating.

Does make toolchains work on your platform?

Yes.

If I set RUST_ENABLE=n and run the Docker build on arm64, I get:

/usr/bin/ccache /optee/build/../toolchains/aarch64/bin/aarch64-linux-ld.bfd -e__ta_entry -pie -T out/ta/os_test/ta.lds -Map=out/ta/os_test/5b9e0e40-2636-11e1-ad9e-0002a5d5c51b.map --sort-section=alignment -z max-page-size=4096  --as-needed   --dynamic-list out/ta/os_test/dyn_list --eh-frame-hdr out/ta/os_test/init.o out/ta/os_test/os_test.o out/ta/os_test/ta_entry.o out/ta/os_test/test_float_subj.o out/ta/os_test/cxx_tests.o out/ta/os_test/cxx_tests_c.o out/ta/os_test/attestation.o out/ta/os_test/user_ta_header.o -L/optee/out-br/build/optee_test_ext-1.0/ta/os_test_lib/out/ta/os_test_lib -los_test -ldl -L/optee/optee_os/out/arm/export-ta_arm64/lib --start-group -lutils -lutee -lmbedtls -ldl /optee/toolchains/aarch64/bin/../lib/gcc/aarch64-buildroot-linux-gnu/12.3.0/../../../../aarch64-buildroot-linux-gnu/lib/../lib64/libstdc++.a /optee/toolchains/aarch64/bin/../lib/gcc/aarch64-buildroot-linux-gnu/12.3.0/libgcc_eh.a --end-group /optee/toolchains/aarch64/bin/../lib/gcc/aarch64-buildroot-linux-gnu/12.3.0/libgcc.a -lutils -o out/ta/os_test/5b9e0e40-2636-11e1-ad9e-0002a5d5c51b.elf
/optee/build/../toolchains/aarch64/bin/aarch64-linux-ld.bfd: /optee/toolchains/aarch64/bin/../lib/gcc/aarch64-buildroot-linux-gnu/12.3.0/libgcc_eh.a(unwind-dw2-fde-dip.o): in function `_Unwind_Find_FDE':
/optee/out-aarch64-sdk/build/host-gcc-final-12.3.0/build/aarch64-buildroot-linux-gnu/libgcc/../../../libgcc/unwind-dw2-fde-dip.c:512: undefined reference to `_dl_find_object'

C++ support in TAs does not work with GCC 12.3. "make toolchains" is supposed to install 11.3, how does your setup end up pulling something from host-gcc-final-12.3.0?
Try adding WITH_CXX_TESTS=n to the make command.

make[4]: *** [/optee/optee_os/out/arm/export-ta_arm64/mk/link.mk:123: out/ta/os_test/5b9e0e40-2636-11e1-ad9e-0002a5d5c51b.elf] Error 1
make[3]: *** [/optee/out-br/build/optee_test_ext-1.0/ta/Makefile.gmake:61: ta-os_test] Error 2
make[2]: *** [package/pkg-generic.mk:284: /optee/out-br/build/optee_test_ext-1.0/.stamp_built] Error 2
make[1]: *** [Makefile:23: _all] Error 2
make[1]: Leaving directory '/optee/out-br'
make: *** [common.mk:341: buildroot] Error 2
The command '/bin/sh -c make V=1 RUST_ENABLE=n' returned a non-zero code: 2

[jforissier]

OPTEE_RUST_ENABLE

That should be RUST_ENABLE=n

This documentation page shows OPTEE_RUST_ENABLE : https://github.com/OP-TEE/optee_docs/blob/9841a9fc14ed49b084bed4b6fd8af417a45cbf24/building/optee_with_rust.rst#L41 That's why I was trying that name.

Ah, yes. The doc needs updating.

OP-TEE/optee_docs#240

[msgilligan]

C++ support in TAs does not work with GCC 12.3. "make toolchains" is supposed to install 11.3, how does your setup end up pulling something from host-gcc-final-12.3.0?

I'm not sure. The only difference in my setup (I'm running a slightly modified version of the example Dockerfile) should be that I'm on arm64.

Try adding WITH_CXX_TESTS=n to the make command.

Will do.

By the way, with the following make line the build completes successfully:

RUN make -j$(nproc) RUST_ENABLE=n arm-tf optee-os qemu

(So disabling Rust and removing the buildroot and linux targets)

[msgilligan]

This works as well:

RUN make -j$(nproc) RUST_ENABLE=n WITH_CXX_TESTS=n

and so does:

RUN make -j$(nproc) RUST_ENABLE=n WITH_CXX_TESTS=n check

[msgilligan]

I opened Issue #749 for the specific problem of the Rust toolchain not being downloaded on aarch64 hosts and I've submitted a "draft" patch that I believe addresses the issue. And PR #748 seems to solve the download part of the issue, but there is still a separate problem with actually building the Rust examples.

I'll leave this issue open because I think we should probably also make an issue for the C++ tests not working on aarch64 hosts.

[msgilligan]

@jforissier asks:

C++ support in TAs does not work with GCC 12.3. "make toolchains" is supposed to install 11.3, how does your setup end up pulling something from host-gcc-final-12.3.0?

Looking at toolchain.mk it says:

# There isn't any native aarch64 toolchain released from Arm and buildroot
# doesn't support distribution toolchain [1]. So we are left with no choice
# but to build buildroot toolchain from source and use it.
#
# [1] https://buildroot.org/downloads/manual/manual.html#_cross_compilation_toolchain

So that seems to answer the question. So maybe we need to adjust the build root configuration to pull/build the compatible version?

Update: I created Issue #751 for this specific problem as there are at least three sub-issues we've found on this issue.

[msgilligan]

I have been unable to reproduce the original error (which I saw on x86_64) reported in this issue, but there are three aarch64 sub-issues now:

• Issue toolchain.mk: Rust toolchain is not downloaded on aarch64 hosts #749, with a fix in PR Download Rust Toolchain during "make toolchains" on aarch64 hosts #748
• Issue C++ tests fail on aarch64 hosts #751 (fixed by PR Downgrade buildroot-built GCC toolchain from 12.x to 11.x #753)
• Issue Compilation of Rust examples fails on aarch64 hosts #752 (no fix yet)

I also opened apache/incubator-teaclave-trustzone-sdk#135 in the Teaclave Rust repository that references these issues, so the Rust developers are made aware -- maybe they have some ideas.

We should probably do one or more of the following:

• Close this issue
• Rename this issue to be a parent issue for aarch64 build issues
• Create a new issue as a parent/catch-all for aarch64 build issue

@jforissier What do you suggest?

[jforissier]

Let's rename this issue to be a parent for the three you mentioned above. Note that I proposed #753 as a fix for #751. This leaves us with #752 to be investigated and it's good that you have asked for help in the teaclave project.

[msgilligan]

Let's rename this issue to be a parent for the three you mentioned above.

Done.

[github-actions]

This issue has been marked as a stale issue because it has been open (more than) 30 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 5 days. Note, that you can always re-open a closed issue at any time.

[msgilligan]

Not stale! We need to fix this!

[jenswi-linaro]

@msgilligan, what's the current status now that #764 has been merged?

[msgilligan]

This comment identified 3 aarch64-host-related issues. At this point, only the last of those issues remains: Issue #752 in this repo (I also created #747 (comment) over on the Rust project to make sure they are aware of it)

Please see apache/incubator-teaclave-trustzone-sdk#135 (comment) which links to a gist with a Containerfile created that works on AMD64 but fails on ARM64.

I am hoping someone with more experience than me with Rust cross-compilation and the custom linking options needed for TAs could take a look at it.

[jenswi-linaro]

Thanks for the update.

[msgilligan]

@msgilligan, what's the current status now that #764 has been merged?

I'm feeling like I misread this question 2 weeks ago. I don't think I was aware of that particular PR and thought you were asking about a previous one that was merged. It looks like there have been multiple commits related to this issue merged, so I will try to build with the latest master and let you know what happens.

[msgilligan]

OK, I've verified that the current master compiles successfully on aarch64 Linux (Debian 12 inside QEMU on macOS.)

But if I set RUST_ENABLE=y then the build fails with the same error as before:

note: /home/sean/ValidiTEE/optee-qemu8/build/../toolchains/aarch64/bin/aarch64-linux-ld.bfd: cannot find -lgcc: No such file or directory

I suppose we could close this issue now that we're down to apparently just #752. Although we might want to open another issue for getting the C++ tests working on aarch64.

(Personal note: I am trying to develop a TA in Rust, so personally I'm much more concerned about the Rust build issue than the C++ one.)

[aElizaF]

I'm attempting to build on a Rpi3 for research purposes. Everything is great until I run the command make toolchains. I noticed that my aarch32 and aarch64 folders do not populate with the proper files. My error seems related to what's happening here.

I'm receiving errors related to aarch64 that wasn't solved by altering the Rust value, like what is suggested above. Any help is appreciated.

configure: error: in /home/user/optee-project/out-aarch64-sdk/build/host-tar-1.35': configure:error: could not enable timestamps after mid-January 2038.

I don't plan on using this build past 2038, so unless necessary I won't be fixing.

But I also receive the following errors:
make[2]: *** [package/pkg-generic.mk:273: /home/user/optee-project/out-aarch64-sdk/build/host-tar-1.35/.stamp_configured] Error 1 make[1]: *** [Makefile:23: _all] Error 2 make[1]: Leaving directory /home/user/optee-project/out-aarch64-sdk make: *** [toolchain.mk:161: /home/user/optee-project/build/.../toolchains/aarch64/.done] Error 2

[msgilligan]

I'm attempting to build on a Rpi3 for research purposes. Everything is great until I run the command make toolchains.

Which OS are you using? Is it a 32-bit or 64-bit version? What is the list of packages you have installed with apt get or equivalent.

Most OP-TEE developers cross-compile on AMD64 (x86) linux. I've been compiling on ARM64 in a Debian VM on Apple Silicon. Building on a Pi 3 will be very slow, but I think we should try to fix any issues that occur. If you have a Pi 4 or Pi 5, I would try using that for the build simply because it will be faster.

[aElizaF]

@msgilligan I followed OP-TEE Prerequisits

I've got a Pi5, but used a 3 because the docs indicate that is what is supported. My Pi3 is armv7l and 32-bit.

[msgilligan]

Which OS are you using? Is it a 32-bit or 64-bit version?

@aElizaF You didn't fully answer this question. Are you using Raspberry Pi OS, Debian, etc? Raspberry Pi OS comes in 32-bit and 64-bit versions (with the 32-bit version being recommended for "most users" by Raspberry Pi) can you let us know whether you are running the 32-bit or 64-bit version and whether your using "bookworm" or "bullseye", etc. I strongly recommend using the 64-bit Bookworm version for builds. (And running it on your Pi5 which is faster and has more memory.)

I have successfully built an OP-TEE disk image for rpi3 using Debian Aarch64 Bookworm in a VM on my MacBook Pro. You do not need to run the build on the same machine you want to run OP-TEE on. Use an x86_64 Linux if you have one, or (after making sure you have 64-bit Raspberry Pi OS installed) try it on your Pi5.

The sequence I used to complete the build was:

 make -j2 toolchains
 make -j `nproc`
 make sdcard-image

Note that I had two issues getting this to work under Debain Aarch64:

1. I had to make the following patch to trusted-firmware-a/Makefile:

diff --git a/Makefile b/Makefile
index 73007b4..4bc058c 100644
--- a/Makefile
+++ b/Makefile
@@ -440,7 +440,7 @@ TF_LDFLAGS          +=      $(subst --,-Xlinker --,$(TF_LDFLAGS_$(ARCH)))
 
 # LD = gcc-ld (ld) or llvm-ld (ld.lld) or other
 else
-TF_LDFLAGS             +=      --fatal-warnings -O1
+TF_LDFLAGS             +=      --no-fatal-warnings -O1
 TF_LDFLAGS             +=      --gc-sections
 # ld.lld doesn't recognize the errata flags,
 # therefore don't add those in that case

There is probably a better fix than making warnings non-fatal, but this change got the build working (and the SDCard image I build boots and passes xtest)

2. I had to add /usr/sbin to my PATH variable so make sdcard-image could find fdisk and mkdosfs:

export PATH=$PATH:/usr/sbin

[msgilligan]

I created a new issue with the 3 fixes/workarounds for building for rpi3 on Debian Aarch64: #775

[msgilligan]

My Pi3 is armv7l and 32-bit.

I just re-read this and I think this means you're running a 32-bit OS. The Pi3 has a 64-bit capable processor, but I'm guessing you're running 32-bit OS and used uname -a or something like that to get the armv7l value.

[aElizaF]

My apologies, I'm using raspbian on the 3. I'll build out the rpi5. I only used the 3 since that was indicated as having support and the 5 didn't. I'll report back in a few days - I haven't been to my lab, so I haven't worked on this.

[aElizaF]

@msgilligan The Rpi5 Works! I used Ubuntu 24.04, and realized I need a bigger storage, as the 64GB ran out of space. I didn't realize OPTEE needed that much disk space. Once I get a larger storage, I'll continue on.

[jbech-linaro]

I didn't realize OPTEE needed that much disk space.

OP-TEE OS un-stripped is less than 5MB and the TA's are very small as well. Usually a few hundreds of kB. Driver for it and user space client code, tee-supplicant adds a few more MB. So, it's not "OP-TEE" that takes disk space.

When it comes to the disk space used when building our reference builds, they tend to vary a bit. Both my QEMU v8 and RPi3 lands on 15GB each when everything has been built. It's typically buildroot, toolchain(s), Linux kernel that takes most disk space.

[github-actions]

This issue has been marked as a stale issue because it has been open (more than) 30 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 5 days. Note, that you can always re-open a closed issue at any time.