Move nixpkgs-check-by-name to a separate repository

[infinisil]

nixpkgs-check-by-name is the tool used for the pkgs/by-name CI check. Its source code is defined in Nixpkgs itself under pkgs/test/nixpkgs-check-by-name.

While it's somewhat convenient for it to be in the same repository it checks, experience has shown that it's really a separate project, especially after #281374, which started pinning the tooling.

It would be best if this codebase was moved to a separate repository, with benefits such as:

• A separate issue/PR tracker, making it easy to track issues and watch for updates. Currently I'm using a milestone to do that instead.
• Improves clarity on the update process, because currently there's always two Nixpkgs PRs needed to apply a change (the first to change the codebase, the second to update the pinned CI tooling)
• A separate Hydra jobset can significantly shorten turnaround time, since there's no need to wait for the NixOS channel to update anymore (that's what is currently used to avoid having to build the tooling for every CI run triggered in GitHub Actions).
• Versioning the tool properly improves signalling regarding the significance of updates.

With more details:

• Move pkgs/test/nixpkgs-check-by-name (except ./scripts) to something under https://github.com/NixOS
  • This codebase is showing promise to also be useful for non-pkgs/by-name checks, so a name not specific to that might be best
  • @philiptaron suggested nixpkgs-vet, since it "vets" the Nixpkgs Architecture :)
• Create a separate Hydra project for the codebase and set up the Nixpkgs-side pinning to use it
  • It's not necessary to expose the tooling's Nix derivation in Nixpkgs, because pinning for CI is best based directly on the store path, because it makes CI very fast because it doesn't need to download any Nixpkgs
• Make at least me the owner of the repository

[philiptaron]

How can I help move this issue along? I find contributing to the tool in nixpkgs difficult, but if it were extracted as nixdoc is extracted, it would be substantially easier.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixpkgs-architecture-team-conclusion-and-prospective/41020/6

[infinisil]

@philiptaron I just asked @zimbatm to create a repo for this so I can do the move. I'd like to just use github.com/NixOS/nixpkgs-check-by-name for now since that's what it's already called, let's have minimal changes at a time, and we can always easily rename it later if it becomes more generic :)

[zimbatm]

Done

[philiptaron]

I look forward to contributing there. It's a little too hard to do it in the giant monorepo, I find. @infinisil's skill in doing so is admirable.

[infinisil]

Steps to complete this:

• Create https://github.com/NixOS/nixpkgs-check-by-name
• Extract all the directories commits into the new repo
  • Extracted history from nixpkgs/pkgs/test/nixpkgs-check-by-name
  • Imported as nixpkgs-check-by-name history
• Set up branch protection on the main branch, making PRs a requirement
• Add @philiptaron to the @NixOS/nixpkgs-check-by-name team, giving write access to the repo
• Transfer issues
• Fix the repo to work on its own
• Set up a Hydra project to build from the new repo
  We're switching to using GitHub releases instead: Automated releases nixpkgs-vet#16
• Switch Nixpkgs CI to pin releases of the new repo: Switch the pkgs/by-name check to a separate repository #297901
• Remove pkgs/test/nixpkgs-check-by-name (except ./script) from Nixpkgs: tests.nixpkgs-check-by-name: Remove now that a separate repo is used #299301

[infinisil]

Just opened #297901 to complete the move. Still a draft yet though