Binary compiled with dontStrip = true crashes

[simonzkl]

I was trying to add debug symbols to my release binary and I noticed the binary compiled with crane and dontStrip = true crashes on startup.

Example project: https://github.com/simonzkl/crane-dont-strip

Steps to reproduce:

1. nix run -L
2. Process exits with Killed: 9

Using cargo build in nix-shell works fine:

1. nix develop -c -- cargo run --release
2. Application exits with exit code 0

Note that this only seems to happen when you link with external dependencies like openssl.

Setting doNotRemoveReferencesToVendorDir = true fixes the issue, so it looks like crane patches something that makes MacOS freak out.

I only tested this on aarch64-darwin.

[dpc]

Hey. We have hit the same issue. I don't think it's a crane issue though. Probably some weird-o-security thing on Macs.

I vaguely recall that upgrading rustc fixed it too.

[simonzkl]

Possibly something to do with codesigning on darwin? I've seen a bunch of places were this was handled in nixpkgs, but I don't really know more than that.

Rust is rustc 1.72.0 in the example above. But I also ran into this issue with 1.73 from rust-overlay.

[dpc]

Possibly something to do with codesigning on darwin?

Yes. Something like that. I'm not an expert in unfree OSes, but I had to deal with SIPs and other problems around it in the past.

Rust is rustc 1.72.0 in the example above.

I might have been wrong.

Seems like we still have

dontStrip = !pkgs.stdenv.isDarwin; 

[simonzkl]

Setting doNotRemoveReferencesToVendorDir = true fixes the issue, so it looks like crane patches something it shouldn't, which was previously stripped out by dontStrip = false. Will update the issue.

[dpc]

Ooohhhhh. I didn't know that. OK then. Crane's fault. I think crane is using binary substitution to override some debug paths in the final binary files that would link the binaries with vendored source code, making Nix consider vendored source code a Nix dependency for your binaries.

I guess it trips some integrity checks / checksums somewhere?

Oh that might be a PITA to undo. THough maybe crane should just cp -aL these files into the build env, at least on MacOS / optionally?

[simonzkl]

Ok I can confirm it's just a signing issue. I added this to my packageBuild:

postFixup = ''
  remove-references-to -t ${pkgs.hello} $out/bin/*
'';

This triggers the codesigning hook here: https://github.com/NixOS/nixpkgs/blob/5dcafc36852cd8d96127f5cbfe42918bf262214e/pkgs/build-support/remove-references-to/default.nix#L27-L29

So we basically just have to replicate this in removeReferencesToVendoredSourcesHook.

[dpc]

Oh. So maybe it's just a matter of order of operations? Does crane need to do it before signing happens?

[simonzkl]

It doesn't seem so. I ran the build with set -x and I don't see any signing going on at all right now. So maybe we just have to set it up ourselves, like removeReferencesTo does.

[simonzkl]

Fyi, to fix this you just have to source ${signingUtils} and call signIfRequired "${installedFile}" in removeReferencesToVendoredSources if the target platform is aarch64-darwin. I attempted this but I don't really understand how do this idiomatically since crane doesn't inherit almost anything from pkgs into its scope. I would need to access pkgs.darwin.signingUtils and pkgs.stdenv somehow.

[dpc]

doesn't inherit almost anything from pkgs into its scope.

I thought the scope inherited all of content of pkgs automatically. But if I'm wrong, you can just add whatever you need to the scope anyway.

[simonzkl]

Nevermind, it seems everything is already there. I just got confused because signingUtils seems to be available in the scope of nixpkgs's callPackage (example), but not here. I'll prepare a PR in a few minutes.

[simonzkl]

Sorta unrelated and not really a crane issue imo, but I did notice that with dontStrip = true binaries tend to leave behind references to build-time dependencies like gcc and rust-std which significantly increases the closure size of the output. This is especially common with statically built C dependencies (via cargo). You can work around this issue with separateDebugInfo = true, but it's not as convenient for debugging.

[ipetkov]

@simonzkl feel free to update the docs or the FAQs with that bit if info!

[dpc]

Promoted to separated issue #428 even if just for visibility and search engines.