Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add plugin package for embedding nix-snapshotter in containerd #112

Merged
merged 2 commits into from
Feb 14, 2024

Conversation

elpdt852
Copy link
Collaborator

@elpdt852 elpdt852 commented Feb 14, 2024

Due to rootless k3s being in its own namespaces (via rootlesskit), it's not possible for an external rootless containerd to be set up with rootless k3s. The only way forward is adding support for nix-snapshotter to k3s upstream by providing a builtin plugin that can be compiled with the containerd embedded in k3s.

This PR provides the plugin package that allows compiling containerd with nix-snapshotter support.

It also includes a commit that has a k3s patch to run with nix-snapshotter support. Since k3s derivation is very difficult to overlay, we include the derivations in-tree to make the changes locally. Once we land support in upstream k3s this can all be deleted.

k3s changes for nix-snapshotter support: hinshun/k3s@9b1247f

@elpdt852 elpdt852 added the ok-to-test Runs NixOS tests label Feb 14, 2024
Copy link
Collaborator

@RobbieBuxton RobbieBuxton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming there isn't a nicer way to copy the code in /k3s/builder.nix from here because it's nix code right? It's annoying because the diff is so small. Maybe we should add a comment linking to where we copied it from? Otherwise happy with the rest of the PR!

@elpdt852
Copy link
Collaborator Author

elpdt852 commented Feb 14, 2024

We can depend on ${nixpkgs}/.../path/to/builder.nix but in general it's bad practice because whenever nixpkgs bumps (even with unrelated changes), the input hash becomes different and k3s needs to be rebuilt from scratch again (it's pretty slow). Ideally we should do it in an overlay, but I tried and it was exceedingly difficult especially when you modify go.mod in the patch because you also need to rebuild the go-modules vendor directory.

In my opinion, since this is short-lived anyway, I'd like to keep it dead simple and just copy & paste. We should file an issue to nixpkgs about making overriding k3s possible.

@elpdt852
Copy link
Collaborator Author

The changes to k3s I've maintain in a fork here: hinshun/k3s@9b1247f

We should avoid reviewing it in the fork or the patch in this PR, instead review it in the upstream k3s PR: k3s-io/k3s#9319

@elpdt852 elpdt852 force-pushed the feature/plugin branch 2 times, most recently from 3de51b9 to e4ecb07 Compare February 14, 2024 14:06
@elpdt852 elpdt852 merged commit ee73e9b into main Feb 14, 2024
7 checks passed
@elpdt852 elpdt852 deleted the feature/plugin branch February 14, 2024 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ok-to-test Runs NixOS tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants