Swapped to containerd from docker and disk usage has doubled - how to fix?

[eradnab]

k3s 1.29.3.

All comparisons are on two freshly installed identical CentOS VMs. The only difference is the docker/containerd setting.

I was using --docker when starting k3s, coupled with docker load. Testing performance I found that not using --docker (and thus containerd) with /usr/local/bin/k3s ctr images import gave me a needed small performance increase. Also containerd seems to be the standard nowadays.

What I've found is that /var/lib/rancher/k3s/agent/containerd is taking twice the space that /var/lib//var/lib/docker/overlay2 used to take.

Using docker:

00:00:44  Filesystem                      Size  Used Avail Use% Mounted on
00:00:44  /dev/mapper/vg_main-lv_var     20G  8.7G   10G  47% /var

Using containerd:

00:00:49  /dev/mapper/vg_main-lv_var     20G   17G  2.3G  88% /var

Using du on /var/lib/ is about 17G on either. But I think du is counting things twice. Using du -x shows docker at 8G and containerd at 14G.

This doubling of space is a problem where I am using k3s where I can't increase the disk space.

Any ideas please?

[brandond]

You might try allowing the image store to discard the unpacked layers?

You'll need to provide your own containerd.tomp.tmpl, as described in the docs:

• https://docs.k3s.io/advanced#configuring-containerd

• k3s/pkg/agent/templates/templates_linux.go

  Lines 10 to 119 in d973fad

  	{{- /* */ -}}
  	# File generated by {{ .Program }}. DO NOT EDIT. Use config.toml.tmpl instead.
  	version = 2
  	[plugins."io.containerd.internal.v1.opt"]
  	path = "{{ .NodeConfig.Containerd.Opt }}"
  	[plugins."io.containerd.grpc.v1.cri"]
  	stream_server_address = "127.0.0.1"
  	stream_server_port = "10010"
  	enable_selinux = {{ .NodeConfig.SELinux }}
  	enable_unprivileged_ports = {{ .EnableUnprivileged }}
  	enable_unprivileged_icmp = {{ .EnableUnprivileged }}
  	{{- if .DisableCgroup}}
  	disable_cgroup = true
  	{{end}}
  	{{- if .IsRunningInUserNS }}
  	disable_apparmor = true
  	restrict_oom_score_adj = true
  	{{end}}
  	{{- if .NodeConfig.AgentConfig.PauseImage }}
  	sandbox_image = "{{ .NodeConfig.AgentConfig.PauseImage }}"
  	{{end}}
  	{{- if .NodeConfig.AgentConfig.Snapshotter }}
  	[plugins."io.containerd.grpc.v1.cri".containerd]
  	snapshotter = "{{ .NodeConfig.AgentConfig.Snapshotter }}"
  	disable_snapshot_annotations = {{ if eq .NodeConfig.AgentConfig.Snapshotter "stargz" }}false{{else}}true{{end}}
  	{{ if .NodeConfig.DefaultRuntime }}default_runtime_name = "{{ .NodeConfig.DefaultRuntime }}"{{end}}
  	{{ if eq .NodeConfig.AgentConfig.Snapshotter "stargz" }}
  	{{ if .NodeConfig.AgentConfig.ImageServiceSocket }}
  	[plugins."io.containerd.snapshotter.v1.stargz"]
  	cri_keychain_image_service_path = "{{ .NodeConfig.AgentConfig.ImageServiceSocket }}"
  	[plugins."io.containerd.snapshotter.v1.stargz".cri_keychain]
  	enable_keychain = true
  	{{end}}
  	{{ if .PrivateRegistryConfig }}
  	{{ if .PrivateRegistryConfig.Mirrors }}
  	[plugins."io.containerd.snapshotter.v1.stargz".registry.mirrors]{{end}}
  	{{range $k, $v := .PrivateRegistryConfig.Mirrors }}
  	[plugins."io.containerd.snapshotter.v1.stargz".registry.mirrors."{{$k}}"]
  	endpoint = [{{range $i, $j := $v.Endpoints}}{{if $i}}, {{end}}{{printf "%q" .}}{{end}}]
  	{{if $v.Rewrites}}
  	[plugins."io.containerd.snapshotter.v1.stargz".registry.mirrors."{{$k}}".rewrite]
  	{{range $pattern, $replace := $v.Rewrites}}
  	"{{$pattern}}" = "{{$replace}}"
  	{{end}}
  	{{end}}
  	{{end}}
  	{{range $k, $v := .PrivateRegistryConfig.Configs }}
  	{{ if $v.Auth }}
  	[plugins."io.containerd.snapshotter.v1.stargz".registry.configs."{{$k}}".auth]
  	{{ if $v.Auth.Username }}username = {{ printf "%q" $v.Auth.Username }}{{end}}
  	{{ if $v.Auth.Password }}password = {{ printf "%q" $v.Auth.Password }}{{end}}
  	{{ if $v.Auth.Auth }}auth = {{ printf "%q" $v.Auth.Auth }}{{end}}
  	{{ if $v.Auth.IdentityToken }}identitytoken = {{ printf "%q" $v.Auth.IdentityToken }}{{end}}
  	{{end}}
  	{{ if $v.TLS }}
  	[plugins."io.containerd.snapshotter.v1.stargz".registry.configs."{{$k}}".tls]
  	{{ if $v.TLS.CAFile }}ca_file = "{{ $v.TLS.CAFile }}"{{end}}
  	{{ if $v.TLS.CertFile }}cert_file = "{{ $v.TLS.CertFile }}"{{end}}
  	{{ if $v.TLS.KeyFile }}key_file = "{{ $v.TLS.KeyFile }}"{{end}}
  	{{ if $v.TLS.InsecureSkipVerify }}insecure_skip_verify = true{{end}}
  	{{end}}
  	{{end}}
  	{{end}}
  	{{end}}
  	{{end}}
  	{{- if not .NodeConfig.NoFlannel }}
  	[plugins."io.containerd.grpc.v1.cri".cni]
  	bin_dir = "{{ .NodeConfig.AgentConfig.CNIBinDir }}"
  	conf_dir = "{{ .NodeConfig.AgentConfig.CNIConfDir }}"
  	{{end}}
  	{{- if or .NodeConfig.Containerd.BlockIOConfig .NodeConfig.Containerd.RDTConfig }}
  	[plugins."io.containerd.service.v1.tasks-service"]
  	{{ if .NodeConfig.Containerd.BlockIOConfig }}blockio_config_file = "{{ .NodeConfig.Containerd.BlockIOConfig }}"{{end}}
  	{{ if .NodeConfig.Containerd.RDTConfig }}rdt_config_file = "{{ .NodeConfig.Containerd.RDTConfig }}"{{end}}
  	{{end}}
  	[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  	runtime_type = "io.containerd.runc.v2"
  	[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  	SystemdCgroup = {{ .SystemdCgroup }}
  	[plugins."io.containerd.grpc.v1.cri".registry]
  	config_path = "{{ .NodeConfig.Containerd.Registry }}"
  	{{ if .PrivateRegistryConfig }}
  	{{range $k, $v := .PrivateRegistryConfig.Configs }}
  	{{ if $v.Auth }}
  	[plugins."io.containerd.grpc.v1.cri".registry.configs."{{$k}}".auth]
  	{{ if $v.Auth.Username }}username = {{ printf "%q" $v.Auth.Username }}{{end}}
  	{{ if $v.Auth.Password }}password = {{ printf "%q" $v.Auth.Password }}{{end}}
  	{{ if $v.Auth.Auth }}auth = {{ printf "%q" $v.Auth.Auth }}{{end}}
  	{{ if $v.Auth.IdentityToken }}identitytoken = {{ printf "%q" $v.Auth.IdentityToken }}{{end}}
  	{{end}}
  	{{end}}
  	{{end}}
  	{{range $k, $v := .ExtraRuntimes}}
  	[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."{{$k}}"]
  	runtime_type = "{{$v.RuntimeType}}"
  	[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."{{$k}}".options]
  	BinaryName = "{{$v.BinaryName}}"
  	SystemdCgroup = {{ $.SystemdCgroup }}
  	{{end}}

Add a line in the [plugins."io.containerd.grpc.v1.cri".containerd] table, setting discard_unpacked_layers = true

Ref: https://github.com/containerd/containerd/blob/8317959018015f6a1756ec8cd08be1093fd630a2/docs/cri/config.md?plain=1#L279-L281

[eradnab]

Thanks for the idea. Finally got round to it today while trying other idea before this. Unfortunately it makes no difference at all.

Example numbers - size of /var/lib/rancher/:

Using docker with Docker image format files: 9 GB
Using containerd with Docker image format files: 19 GB
Using containerd with Docker image format files with discard_unpacked_layers = true: 19 GB
Using containerd with OCI image format files: 12 GB (better but not good enough)

Any other ideas to get the disk space down to the same level as docker please?

[brandond]

This is all just containerd behavior, not k3s. You might take your inquiry to the containerd project.

[eradnab]

Yes you are right. I wondered if it was specific to k3s containerd.

Anyway I have an update.

By putting in a go template to change the config for containerd to discard_unpacked_layers = true AND also ensuring the images were in OCI image format the size is back down to 9 GB.

However an even simpler answer was to add:
unzstd $image -c | /usr/local/bin/k3s ctr images import --all-platforms --discard-unpacked-layers -

[brandond]

Settings from the plugins."io.containerd.grpc.v1.cri".containerd section are specific to the CRI service only take effect when using the CRI interface (kubelet or crictl) to interact with containerd - if you're using ctr to import images then yes, you'd need to manually set flags to discard unpacked layers.