incus images based on Ubuntu fail to start

[sbromberger]

Is this a new report?

Yes

System Info

Void 6.6.11_1 x86_64 AuthenticAMD uptodate rFF

Package(s) Affected

incus-0.4.0_1

Does a report exist for this bug with the project's home (upstream) and/or another distro?

No response

Expected behaviour

Migrated from a home-built version of incus to the new package (yay!) - but while void images start up with no problem, ubuntu ones (including new images) fail to start. I expected the ubuntu images to start up normally.

Actual behaviour

Ubuntu images (based on lunar; possibly others) fail to start. There is no diagnostic information printed with -v.

--console produces the following error:

[root@elemental beryllium]# incus start testub --console
To detach from the console, press: <ctrl>+a q
Error: Failed running forkconsole: "container is not running: \"testub\""
                                                                         Error: write /dev/pts/ptmx: file already closed

--debug produces the following output: https://gist.github.com/sbromberger/e6adcff70ab7e6fd0410055cfde09e69

Steps to reproduce

1. incus create images:ubuntu/lunar testub
2. incus start testub
3. incus ls testub
4. Observe that status is STOPPED

[sbromberger]

Ah, I found some logs in /var/log/incus/testub:

Excerpt from lxc.log:

lxc testub 20240124172933.916 INFO     conf - ../src/lxc/conf.c:setup_personality:1917 - Set personality to "0lx0"
lxc testub 20240124172933.917 NOTICE   conf - ../src/lxc/conf.c:lxc_setup:4469 - The container "testub" is set up
lxc testub 20240124172933.918 NOTICE   start - ../src/lxc/start.c:start:2194 - Exec'ing "/sbin/init"
lxc testub 20240124172933.921 NOTICE   start - ../src/lxc/start.c:post_start:2205 - Started "/sbin/init" with pid "15889"
lxc testub 20240124172933.921 NOTICE   start - ../src/lxc/start.c:signal_handler:446 - Received 17 from pid 15890 instead of container init 15889
lxc testub 20240124172933.978 INFO     error - ../src/lxc/error.c:lxc_error_set_and_log:31 - Child <15889> ended on error (255)
lxc testub 20240124172933.995 INFO     conf - ../src/lxc/conf.c:run_script_argv:340 - Executing script "/usr/libexec/incus/incusd callhook /var/lib/incus "default" "testub" stopns" for container "testub"
lxc testub 20240124172934.129 INFO     conf - ../src/lxc/conf.c:lxc_map_ids:3603 - Caller maps host root. Writing mapping directly
lxc testub 20240124172934.130 NOTICE   utils - ../src/lxc/utils.c:lxc_drop_groups:1368 - Dropped supplimentary groups
lxc testub 20240124172934.147 INFO     conf - ../src/lxc/conf.c:run_script_argv:340 - Executing script "/usr/libexec/incus/incusd callhook /var/lib/incus "default" "testub" stop" for container "testub"

Full log here: https://gist.github.com/sbromberger/b05b018f3df1192a0a6bada49d8be1e8

[sbromberger]

I found these logs in console.log on another image as well:

Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...

[CameronNemo]

As mentioned in the PR, does this happen with the unified cgroup mode i.e. a cgroup2-only setup?

The latest systemd release actually deprecated cgroup v1 support, and it will be removed in a future systemd release:

• We intend to remove cgroup v1 support from a systemd release after
  the end of 2023. If you run services that make explicit use of
  cgroup v1 features (i.e. the "legacy hierarchy" with separate
  hierarchies for each controller), please implement compatibility with
  cgroup v2 (i.e. the "unified hierarchy") sooner rather than later.
  Most of Linux userspace has been ported over already.

https://github.com/systemd/systemd/releases/tag/v255

[BikyAlex]

Confirm this happens to other distros. I tested Debian and NixOS, both fail to start with the same operation not permitted / failed to mount API FS error.

I tested Incus on a NixOS host and there systemd distros launch fine. Not sure why this is an issue on Void alone.

Anyway, edited rc.conf and set "CGROUP_MODE=unified" and rebooted. Debian and NixOS start now. Not sure what's the point of keeping cgroup in hybrid by default in Void, if any other things depend on that.

Well, I'm happy I can use Incus with this workaround. I wanted to be able to run stuff on other distros in case there's no package in Void.

[CameronNemo]

@BikyAlex I am not entirely sure how to escalate the "let's move CGROUP_MODE to unified by default" pitch, but I definitely don't think anything really depends on the hybrid cgroups anymore. Except for perhaps older device kernels like rpi kernels?

[classabbyamp]

rpi kernels are on 6.1, not sure what the kconfig sets wrt cgroups though

[CameronNemo]

@classabbyamp

void-linux/void-runit#103 would resolve this issue IMO