Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC 140 2a] pkgs/by-name: Enforce for new packages #275539

Merged
merged 6 commits into from
Jan 15, 2024

Conversation

infinisil
Copy link
Member

@infinisil infinisil commented Dec 20, 2023

Note

An open call for the final review and merge was held on 2024-01-15, see #275539 (comment)

Context

This is part of the effort to implement RFC 140, aka pkgs/by-name. While previous PRs already enabled users to define packages using pkgs/by-name, there's still thousands of packages still using all-packages.nix, and new ones added every day.

This PR is part of the RFC 140 migration plan, building on top of #272395 and #274591.

This work is sponsored by Antithesis and Tweag

Changes

This PR prevents new packages using pkgs.callPackage from being defined not in pkgs/by-name. Instead the pkgs/by-name directory has to be used instead. This the A-side of Part 2 of the RFC.

This doesn't affect packages defined using alternate callPackage's like libsForQt5.callPackage or callPackages, which pkgs/by-name cannot handle (yet?).

Note

The effects of this PR will only be seen once the nixos-unstable channel updates, since its pre-built version of the tooling is used for CI.

Things done

  • Tests updated
    • Test to ensure that new packages using callPackage but not using pkgs/by-name throw an error
    • Test to ensure that a package moved out of pkgs/by-name throws an error
    • Test to ensure that new packages using an alternate callPackage doesn't give an error.
  • Tested on the entirety of Nixpkgs
    • Ensured okay performance (about 25 single-core seconds on my machine)
  • Tidy code
  • Clean history

Add a 👍 reaction to pull requests you find important.

@infinisil
Copy link
Member Author

This PR is now theoretically ready, but since the changes got a bit big, I split off everything not directly related to this PR into #278805, which should be merged first.

Comment on lines 6 to 8
Packages found in the named-based structure do not need to be explicitly added to the
Packages found in the name-based structure must not be added to the
`top-level/all-packages.nix` file unless they require overriding the default value
of an implicit attribute (see below).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is still better to lead with the positive part. Something like

Packages found in the name-based structure are automatically included, without needing to be added in a file like all-packages.nix. They must only occur in file if they require overriding the default value of an implicit attribute (see below).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed the wording a little bit, but this is essentially applied now :)

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-01-09-nixpkgs-architecture-team-meeting-47/38037/1

@infinisil infinisil marked this pull request as ready for review January 9, 2024 22:03
pkgs/by-name/README.md Outdated Show resolved Hide resolved
Copy link
Contributor

@philiptaron philiptaron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a thorough look. The tests are quite clear. Nice work. I don't have any code related comments, just copy-editing. Might also run a RFC101 formatter over these Nix files just for giggles.

pkgs/test/nixpkgs-check-by-name/src/nixpkgs_problem.rs Outdated Show resolved Hide resolved
pkgs/test/nixpkgs-check-by-name/src/ratchet.rs Outdated Show resolved Hide resolved
if let Some(path) = &call_package_path {
format!("./{}", path.display())
} else {
"...".into()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pkgs/test/nixpkgs-check-by-name/src/nixpkgs_problem.rs Outdated Show resolved Hide resolved
pkgs/test/nixpkgs-check-by-name/src/eval.rs Outdated Show resolved Hide resolved
Comment on lines +197 to +204
// This is a bit of an odd case: We don't even _know_ whether this attribute
// would qualify for using pkgs/by-name. We can either:
// - Assume it's not using pkgs/by-name, which has the problem that if a
// package evaluation gets broken temporarily, the fix can remove it from
// pkgs/by-name again
// - Assume it's using pkgs/by-name already, which has the problem that if a
// package evaluation gets broken temporarily, fixing it requires a move to
// pkgs/by-name
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Future ratchets might try to drive this number down, I might imagine.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah, I thought about implementing an eval-successful ratchet, but I think this should be a future PR. Then also it's also not pkgs/by-name-specific anymore and the tool could use a rename.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One only needs to look at the list of things detected by NixpkgsProblems to realize we've backed ourselves into a damn fine nixpkgs linter framework. 🚀

pkgs/test/nixpkgs-check-by-name/src/eval.nix Outdated Show resolved Hide resolved
Not that important, but nice.
Also adds a nice test case show-casing the two current ratchet checks at
once.
@infinisil
Copy link
Member Author

infinisil commented Jan 16, 2024

Unfortunately this has a bug that needs to be addressed. Without doing anything, all PRs to master would start failing CI in like 1 day once nixos-unstable updates. I opened this PR to prevent this: #281374

Here's a follow up to fix the bug and improve CI to prevent this in the future: #281390

infinisil added a commit to tweag/nixpkgs that referenced this pull request Jan 16, 2024
infinisil added a commit to tweag/nixpkgs that referenced this pull request Jan 16, 2024
infinisil added a commit to tweag/nixpkgs that referenced this pull request Jan 16, 2024
This was an oversight in NixOS#275539
not accounted for, which would've failed in CI
cvanfleteren pushed a commit to cvanfleteren/nixpkgs that referenced this pull request Jan 17, 2024
This was an oversight in NixOS#275539
not accounted for, which would've failed in CI
@infinisil
Copy link
Member Author

Now that nixos-unstable contains the latest fixes, here's the final follow-up PR: #281835 which starts enabling the enforcement!

SuperSandro2000 pushed a commit to SuperSandro2000/nixpkgs that referenced this pull request Jan 19, 2024
SuperSandro2000 pushed a commit to SuperSandro2000/nixpkgs that referenced this pull request Jan 19, 2024
SuperSandro2000 pushed a commit to SuperSandro2000/nixpkgs that referenced this pull request Jan 19, 2024
infinisil added a commit that referenced this pull request Jan 19, 2024
@h7x4 h7x4 mentioned this pull request Jan 20, 2024
13 tasks
SuperSandro2000 pushed a commit to SuperSandro2000/nixpkgs that referenced this pull request Jan 21, 2024
SuperSandro2000 pushed a commit to SuperSandro2000/nixpkgs that referenced this pull request Jan 21, 2024
mkg20001 pushed a commit to mkg20001/nixpkgs that referenced this pull request Jan 21, 2024
peterhoeg pushed a commit to peterhoeg/nixpkgs that referenced this pull request Jan 22, 2024
@lelgenio lelgenio mentioned this pull request Feb 7, 2024
13 tasks
@arikgrahl arikgrahl mentioned this pull request Feb 12, 2024
13 tasks
infinisil added a commit to NixOS/nixpkgs-vet that referenced this pull request Mar 18, 2024
This was an oversight in NixOS/nixpkgs#275539
not accounted for, which would've failed in CI
infinisil added a commit to NixOS/nixpkgs-vet that referenced this pull request Mar 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants