Gracefully handle dangling revisions

[johnrichardrinehart]

Foreword: I've created #11753 to try to address the below.

Describe the bug

nix versions >=2.20 don't seem to have this problem and I understand that much of the libfetchers/git work has been refactored including a change to use libgit2 instead of git on the $PATH. So, the below only applies to earlier versions of nix.

I've noticed that occasionally my nix derivations fail to evaluate because the bare checkout in the fetcher cache contains dangling revisions which my derivations rely on. This implies that the bare checkout in the fetcher cache is in an unhealthy state so we should fetch the references from the remote to try to update the checkout to a healthy state, I guess. But, I thought I'd call out this bug and share a reproduction. I'll share a patch which redresses this, shortly. You can modify the below script to use that patched version to demonstrate the fix.

I'll note that this was called out by @edolstra as a potential issue when he first pushed these changes. I haven't checked if this has already been reported by other users. It happens very rarely and I think only in cases where the fetch could not be completed successfully (like the evaluator thread crashes or the remote disconnects abruptly or other end-of-the-world type scenarios). The rarity is one reason my script is a little hacky - it's hard to reproduce under normal circumstances.

WARNING: My script will perform destructive operations on both your v1 fetcher cache and fetcher cache database. I've tried to write it such that it will only touch the paths and rows it cares about. But the fact that bugs exist in software is why I'm even here in the first place.

Steps To Reproduce

Run this

#!/usr/bin/env bash

this_nixpkgs='github:nixos/nixpkgs?rev=36ac8d7e411eeb11ac0998d5a39e329c1226e541';
this_git=$(nix build --print-out-paths "$this_nixpkgs#git")/bin/git;

# TODO: investigate why git doesn't allow Unix timestamps smaller than 100000000
export GIT_AUTHOR_NAME=foo \
	[email protected] \
	GIT_AUTHOR_DATE=100000000 \
	GIT_COMMITER_NAME=foo \
	[email protected] \
	GIT_COMMITTER_DATE=100000000;

tmpgit=$(mktemp -d);
echo "Repo'll be at $tmpgit";

"$this_git" init "$tmpgit";

"$this_git" -C "$tmpgit" commit --allow-empty -m "foo"; # 8c2146823865b76da067b3bb458611a0a19ede3b
"$this_git" -C "$tmpgit" commit --allow-empty -m "foo"; # 2da0785fa32ce4c628d501c8743c88be00835b50
"$this_git" -C "$tmpgit" reset --hard HEAD~1; # 8c2146823865b76da067b3bb458611a0a19ede3b


for version in 18 19 20 21 22 23 24; do
    echo "Checking nix version 2.$version";
    this_nix=$(nix build "$this_nixpkgs#nixVersions.nix_2_$version^out" --print-out-paths)/bin/nix;

    url="file:///$tmpgit";

    # populate fetcher-v1.sqlite and the gitv3/ dir
    "$this_nix-build" --expr $'builtins.fetchGit {
        url = \"'"$url"$'\";
        rev = \"8c2146823865b76da067b3bb458611a0a19ede3b\";
        submodules = true;
    }';

    # put the "corrupted" git dir in place of the fetched one
    cache_dir="${XDG_CONFIG_HOME:-$HOME}"/.cache/nix/gitv3;
    # TODO: investigate a regression in 2.20 which no longer supports stdin redirection because it's a symlink forcing me to use another temp file....
    tmp=$(mktemp);
    printf "%s" "$url" >"$tmp";
    fetched_dir="$cache_dir"/$(printf "%s" "$url" | $this_nix --extra-experimental-features nix-command hash file --type sha256 --base32 "$tmp");
    rm -rf "$fetched_dir";
    git clone --bare "$tmpgit" "$fetched_dir";

    # See if we can realise the derivation pointing to the dangling commit
    _NIX_FORCE_HTTP=1 "$this_nix-build" --impure --expr $'
    { pkgs ? (import <nixpkgs> { }) }: pkgs.stdenv.mkDerivation {
      pname = "foo";
      version = "1.0.0";
      dontUnpack = true;

      '"
      src = builtins.fetchGit {
        url = \"file:///$tmpgit\";
        rev = \"2da0785fa32ce4c628d501c8743c88be00835b50\";
        submodules = true;
      };

      buildPhase = ''
        touch \$out;
        exit 0;
      '';
    }
    "
done

Expected behavior

I'm not sure. I guess we should probably check reachability here and we should just fetch the rev (since it must be known to exist at this point) here.

nix-env --version output

The script iterates from 2.18 to 2.24. Sorry, I'm not sure exactly which revs of those versions.

Additional context

Priorities

Not a super high priority. But, it'd be nice not to rely on a forked version of nix.

Please see #11753 for a change set which appears to address this behavior.

Add 👍 to issues you find important.