python3Packages.jaxlib: fetchAttrs reproducibility

[SomeoneSerge]

Description of changes

#321920 (comment) says bazel downloads external/go_sdk/versions.json from an unpinned URL resulting in an unstable FOD hash; "good" news is, the build doesn't seem to actually use any of this golang stuff...

• I don't know if this file was the only source of instability or there are more left.
• I need help getting the hashes for other platforms

CC @lromor @GaetanLepage @samuela @zimbatm

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[SomeoneSerge]

Note: the file has to exist, otherwise bazel-build will attempt a re-download.
Note: the original file looks something like

[
 {
  "version": "go1.23rc1",
  "stable": false,
  "files": [
   {
    "filename": "go1.23rc1.src.tar.gz",
    "os": "",
    "arch": "",
    "version": "go1.23rc1",
    "sha256": "6e9c4765872808663ccf0b937e5637c7df67146943fa6e8909a81a0d129c8045",
    "size": 28156320,
    "kind": "source"
   },
   {
    "filename": "go1.23rc1.aix-ppc64.tar.gz",
    "os": "aix",
    "arch": "ppc64",
    "version": "go1.23rc1",
    "sha256": "d0cfbc6fda8a6fa993e24b42ebe77cd1109fb7200cb315a90ce72d75c78075fc",
    "size": 97662103,
    "kind": "archive"
   },
   {
    "filename": "go1.23rc1.darwin-amd64.tar.gz",

[zimbatm]

alternativley (needs findutils and moreutils)

find -name "*.json" -exec sh -c "jq --sort-keys < {} | sponge {}" \;

[SomeoneSerge]

Not sure we can do that to all of $bazelOut/, but at least to $bazelOut/external/go_sdk? Ideally, I suppose, we'd only leave the bare minimum unmasked, but let's start small:)

[GaetanLepage]

Well done ! You need to provide the missing hashes though.

[zimbatm]

alternativley (needs findutils and moreutils)

find -name "*.json" -exec sh -c "jq --sort-keys < {} | sponge {}" \;

[SomeoneSerge]

Ofborg seems ok, let's give it a try

Future work:

• Check in on this in a week to see if the hash has still mutated again
• Consider a more expansive masking: python3Packages.jaxlib: fetchAttrs reproducibility #323681 (comment)

[vcunat]

Maybe you know off the top of your head what's best for NixOS 24.05? It won't fetch now: https://hydra.nixos.org/build/269853893
(there have been multiple jaxlib changes since 24.05 branch-off)

[SomeoneSerge]

@vcunat I can't immediately see what is the real error in https://hydra.nixos.org/build/269853893/nixlog/3, but if it's the $bazelOut/external/go_sdk/versions.json" again, we should maybe just reapply postPatch regardless of other changes?

[vcunat]

The error is hash mismatch of the fixed-output derivation, I believe.

[SomeoneSerge]

Ok yes it is the hash, and I'd assume it is the same unpinned URL issue. I can backport the change in a few days unless someone else does it first

ss@ubuntu:~$ nix build github:NixOS/nixpkgs/staging-next-24.05#python3Packages.jaxlib-build -j0
error: build of '/nix/store/azrkbn4fp2lr67lm7mg34dc917b3hjwj-bazel-build-jaxlib-0.4.28-deps.tar.gz.drv' on 'ssh://[email protected]' failed: hash mismatch in fixed-output derivation '/nix/store/azrkbn4fp2lr67lm7mg34dc917b3hjwj-bazel-build-jaxlib-0.4.28-deps.tar.gz.drv':
         specified: sha256-P5JEmJljN1DeRA0dNkzyosKzRnJH+5SD2aWdV5JsoiY=
            got:    sha256-8HEKP0ttuaY+Et45+AzIbHCXj97OeMzq5m5sJPF3h2Y=

Side note, I could build jaxlib-build.src directly somehow:

ss@ubuntu:~$ nix build github:NixOS/nixpkgs/staging-next-24.05#python3Packages.jaxlib-build.src -j0
error:
       … while evaluating the flake output attribute 'legacyPackages.aarch64-linux.python3Packages.jaxlib-build.src'

       error: string '/nix/store/kv5apk05w01g67r4cfxd2vs2k3dvrgxp-bazel-build-jaxlib-0.4.28/jaxlib-0.4.28-cp311-cp311-manylinux2014_aarch64.whl' has context with the output 'out' from derivation '/nix/store/rksyil5dgyi5s83mgiwv40ynnialjg90-bazel-build-jaxlib-0.4.28.drv', but the string is not the right placeholder for this derivation output. It should be '/nix/store/kv5apk05w01g67r4cfxd2vs2k3dvrgxp-bazel-build-jaxlib-0.4.28'
ss@ubuntu:~$ nix build github:NixOS/nixpkgs/staging-next-24.05#python3Packages.jaxlib-build -j0