Multi-architecture docker build produces binary that fails to execute due to QEMU lib error

[ptr727]

I ended up at this issue when I could not get my .net 7.0 builds to compile using amd64 and arm64 environments.

I followed the mixture of advice in the issue and this blog post, and I am now building using .NET SDK 8.0 preview nightly builds.

I am at a loss of figuring out why dotnet run will work when cross compiling amd64->arm64, but executing the dotnet publish binary in the same environment fails with QEMU errors, e.g. qemu-aarch64: Could not open '/lib/ld-linux-aarch64.so.1': No such file or directory.

I tested with Docker Desktop for Windows, that I believe has the QEMU bits baked in, and I tested with Github Actions and with QEMU and BuildX configured for cross platform environments (i.e. installing tonistiigi/binfmt).

If I had to guess I'd say it is a problem related to the TARGETARCH producing executables that are incompatible with the binary runtime environment, but I'm guessing.

Here is a failing build run.
Here is the actions file.
Here is the docker file.

Interesting bits:

dotnet run is happy:

#20 [linux/amd64->arm/v7 builder  7/10] RUN dotnet run --project ./PlexCleaner/PlexCleaner.csproj --version
#0 4.608 /root/.nuget/packages/microsoft.build.tasks.git/1.1.1/build/Microsoft.Build.Tasks.Git.targets(25,5): warning : Unable to locate repository with working directory that contains directory '/Builder/PlexCleaner'. [/Builder/PlexCleaner/PlexCleaner.csproj]
#0 4.654 /root/.nuget/packages/microsoft.build.tasks.git/1.1.1/build/Microsoft.Build.Tasks.Git.targets(48,5): warning : Unable to locate repository with working directory that contains directory '/Builder/PlexCleaner'. [/Builder/PlexCleaner/PlexCleaner.csproj]
#0 4.674 /root/.nuget/packages/microsoft.sourcelink.common/1.1.1/build/Microsoft.SourceLink.Common.targets(53,5): warning : Source control information is not available - the generated source link is empty. [/Builder/PlexCleaner/PlexCleaner.csproj]
#0 5.394 1.1.1
#20 DONE 5.5s

dotnet publish is happy:

#22 [linux/amd64->arm/v7 builder  8/10] RUN dotnet publish ./PlexCleaner/PlexCleaner.csproj         --arch arm         --self-contained false         --output ./Build/Release         --configuration release         -property:Version=3.0.0.0         -property:FileVersion=3.0.42.54657         -property:AssemblyVersion=3.0.42.54657         -property:InformationalVersion=3.0.42+d581164e46         -property:PackageVersion=3.0.42-gd581164e46     && dotnet publish ./PlexCleaner/PlexCleaner.csproj         --arch arm         --self-contained false         --output ./Build/Debug         --configuration debug         -property:Version=3.0.0.0         -property:FileVersion=3.0.42.54657         -property:AssemblyVersion=3.0.42.54657         -property:InformationalVersion=3.0.42+d581164e46         -property:PackageVersion=3.0.42-gd581164e46
#22 34.14   PlexCleaner -> /Builder/PlexCleaner/bin/debug/net7.0/linux-arm/PlexCleaner.dll
#22 34.51   PlexCleaner -> /Builder/Build/Debug/
#22 DONE 34.8s

dotnet publish output binary fails to execute:

#27 [linux/amd64->arm/v7 builder 10/10] RUN ./Publish/PlexCleaner/PlexCleaner --version
#0 0.116 qemu-arm: Could not open '/lib/ld-linux-armhf.so.3': No such file or directory
#27 ERROR: process "/bin/sh -c ./Publish/PlexCleaner/PlexCleaner --version" did not complete successfully: exit code: 255

Any ideas?

[ptr727]

@richlander for input please?

[richlander]

Related to this? docker/buildx#1222

In general, we don't support QEMU, but this may be a separate issue, since that error seems to have a fair number of hits.

[ptr727]

I saw that issue in research, but it seems that the particular error just called for correctly installing QEMU (using tonistiigi/binfmt that actions does), and arm64 was stated to work but arm v7 not, where in my case neither work.

Second, the dotnet arm64 binaries from the debian arm64 image execute, so I assume QEMU is working for the debian and dotnet arm64/7 binaries, but the binary compiled by dotnet publish fails to run in the same environment.

As far as generally not supporting QEMU, I though the whole reason for all this work was to have cross compiling with compilation done in the actual docker build such that builder host does not need to be the cross compiler, am I mistaken in my expectation?

[richlander]

The new pattern -- #4388 (comment) -- is intended to:

• Enable the SDK to always run on native-arch.
• Enable cross-targeting.

We are not working on enabling execution (including dotnet run) to work cross-arch within an emulated environment. Certainly, .NET runs on multiple architectures.

[ptr727]

Hmm, ok, I misunderstood the level of support.

So to confirm, the arm64 dotnet binaries (just like the OS) run fine under QEMU.
Binaries compiled for arm64 using arm64 dotnet does NOT support running under QEMU, but will run on actual arm64 hardware.

You said dotnet run is not being worked on, but in my testing dotnet run and dotnet test did work fine on arm64 QEMU?

I now recall some other issues I read where the problem now becomes that of pipeline testing, if I can only test arm64 compiled binaries on arm64 hardware, but I can cross-compile, I still need some way of testing .NET compiled binaries, leaving me to either rely on dotnet test working (is it supported?) not test the compiled binaries for anything other than x64, or to run CI on x64, armv7, and arm64 hardware, not really a multi platform build pipeline.

[richlander]

.NET runs in some scenarios under QEMU. I'm just saying we don't generally support it or define which scenarios work and which don't.

This is the reason for getting the SDK to run under $BUILDPLATFORM. That way the SDK runs NOT under QEMU.

You are right that we don't have a way of running tests under emulation.

FYI: https://gitlab.com/qemu-project/qemu/-/issues/249

[ptr727]

Got it, I (naively) expected both the tools and the compiled output to run while being compiled.
Would be cool to have full QEMU support for compiled binaries (assuming all nuget / platform dependencies also support QEMU), but I now understand what is going on, and I'll stick to compiling but not testing.

[ptr727]

Follow-up, maybe generally useful, is there a way to test (e.g. environment variables) that the current executing platform is natively supported, i.e. let's say I build multi-arch including amd64 that is my host platform, and my dockerfile can if platform = native-platform then run tests?

If such functionality is available I could at least stick with one actions file and one dockerfile, and I do not have to dotnet test in the actions script or in a separate image.

[richlander]

That's a great question. Yes, that could definitely work.

You can merge this pattern with the one used in complexapp.

https://github.com/dotnet/dotnet-docker/blob/main/samples/complexapp/README.md

[ptr727]

I see, I typically fail the build if dotnet test fails, and creating a test entrypoint would also need to test for the "correct" platform, else the same QEMU error when running that image on the wrong platform.

Is there way to get the host OS platform (native where QEMU would not used) from within the Dockerfile, i.e. if my actions run using runs-on: ubuntu-latest I'd expect to get some form of linux/x64, and I could compare the (fake) actions.host.platform (or is this $BUILDPLATFORM) with $BUILDPLATFORM or $TARGETARCH

I would then do something like RUN if [ "build platform" = "target platform" ]; then dotnet test ...

[krylor]

E.g. https://github.com/ptr727/PlexCleaner/blob/multiarch/Docker/Debian.dotNET.Dockerfile

Hello @ptr727

The link above is dead?
I am trying to cross compile for arm-64 Amazon Linux on Windows running on x64, is this even possible?

Regards

[lbussell]

Yes, it should be possible. Our samples are set up for multi-platform builds. If you wanted to cross compile that Dockerfile, you'd just need to pass the --platform linux/arm64 command to your Buildx build.

Are you using Amazon Linux as a host OS or a container OS? I looked at the amazonlinux image from DockerHub, and I saw that its repository only contains .NET 6. If you use that image for deployment, you'll obviously need to modify the runtime image layer to install .NET (or deploy your app as self-contained), and you'll need to compile with the .NET 6 SDK image using the -r linux-$TARGETARCH flag instead of -a $TARGETARCH (that argument is new). Let me know if you have any questions.

EDIT: To clarify further, you might be able to use .NET 8 if you deploy your app as self-contained.

[ptr727]

The link above is dead?
Try this: https://github.com/ptr727/PlexCleaner/blob/main/Docker/Debian.dotNET.Dockerfile