Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Just use rootless Buildkitd to run containers #25

Closed
richiejp opened this issue Aug 22, 2024 · 0 comments · Fixed by #26
Closed

Just use rootless Buildkitd to run containers #25

richiejp opened this issue Aug 22, 2024 · 0 comments · Fixed by #26
Labels
enhancement New feature or request

Comments

@richiejp
Copy link
Contributor

Currently we build containers in buildkit with containerd as the worker, then run the containers with nerdctl on containerd.

This has a few problems:

  1. harder to bundle both containerd and buildkit
  2. long pause when exporting to image for larger images (so that nerdctl can start the container)
  3. harder to get rootless mode to work due to more components being involved

However most of the time we just want to run a program quickly and test it. We don't need to export images. Buildkit of course can run programs in a container. The catch is that it doesn't support networking to the same degree that nerdctl does. For example there doesn't appear to be any interface for the CNI port forwarding plugin. It's not clear if bridge mode is fully supported and probably there is more.

However rootlesskit and slirp4ns can forward ports (https://github.com/rootless-containers/slirp4netns/blob/master/slirp4netns.1.md#api-socket, https://github.com/rootless-containers/rootlesskit/blob/master/docs/port.md) from the namespace that buildkitd runs in and buildkitd defaults to using the host's network in rootless mode (meaning the network namespace created by rootlesskit, so not really the host's network).

For the purposes of running a single app in a quick feedback loop this is good enough. It will be faster, won't require root and should be easier to bundle because we just have one daemon to start. The downside is that for running multiple apps we need to consider port collisions which may require a more complicated CNI bridge mode setup.

So the plan is:

  1. Start buildkitd at startup with rootlesskit and slirp4netns
    Initially with something like rootlesskit --port-driver=builtin -p 127.0.0.1:5000:5000/tcp --net=slirp4netns --copy-up=/etc --disable-host-loopback
  2. Instead of using nerdctl to run the application, run it at the end of the build process in a buildkit build/solve call.
  3. Bundle buildkitd, rootlesskit, slirp4netns and whatever else into Ayups executable.

Curiously the Buildkit gateway client has a NewContainer call which we can then Start a process in. This doesn't appear to be part of the build process, it's just a way to run a container.

If this works then it solves #18

We probably still want to support external buildkitd and Containerd daemons for k8s compatibility, but it's less important.

@richiejp richiejp added the enhancement New feature or request label Aug 22, 2024
@richiejp richiejp linked a pull request Aug 23, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant