failed to initialize top level QOS containers: root container [kubepods] doesn't exist

[slapcat]

Summary

Kubelet fails to start after bootstrap completes. The system journal shows the following error from the snap.k8s.kubelet.service unit:

kubelet.go:1547] "Failed to start ContainerManager" err="failed to initialize top level QOS containers: root container [kubepods] doesn't exist"

What Should Happen Instead?

Kubelet is configured correctly to start on the system.

Reproduction Steps

1. sudo snap install k8s --channel=latest/edge --classic
2. sudo k8s bootstrap
3. sudo systemctl status snap.k8s.kubelet.service

System information

# cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy

Can you suggest a fix?

This issue might be related to the upstream bug here: cri-o/cri-o#7701

Adding these lines to the bottom of /var/snap/k8s/common/args/kubelet and restarting snap.k8s.kubelet.service fixes the issue:

--cgroups-per-qos=false
--enforce-node-allocatable=""

Are you interested in contributing with a fix?

No response

[neoaggelos]

Hi @slapcat, thank you for bringing this up.

See canonical/microk8s#4361 (comment) for the related issue in MicroK8s. Though these args can help to work around the bug, in MicroK8s we went the route of adjusting the kubelet service so that the required cgroup controllers are properly delegated, see canonical/microk8s#4503

We will need to adjust that patch for k8s-snap as well.

[bschimke95]

Hi @slapcat,

#550 should solve this issue.
Please try with the fix and close this issue if this works for you.