Fedora CoreOS 41 breaks k8s nodes

[craigcabrey]

https://discussion.fedoraproject.org/t/fedora-coreos-rebasing-to-fedora-linux-41/134786

ComposeFS is the rootfs now, which looks like there is no free space left / read only file system. Then, k8s refuses to schedule onto the node due to this.

[brandond]

This sounds like a coreos or Kubernetes issue, not k3s. Put the Kubernetes bits on a traditional writable filesystem, like ext4 or xfs.

[craigcabrey]

Yea, it probably is.

Put the Kubernetes bits on a traditional writable filesystem, like ext4 or xfs

This seems like an odd statement, the installation is on a writeable filesystem. It is / specifically that is marked read only, which k8s interprets as no usable space even though the working directory does have usable (writeable) space.

All major platforms are moving towards an immutable paradigm, so I don't think it's as simple as "don't use that".

[brandond]

the working directory does have usable (writeable) space.

What do you mean by "working directory". Kubernetes (and k3s) need writable filesystems in several places in the host, and at least one of those needs to support overlayfs.

All major platforms are moving towards an immutable paradigm

That's great. You still need writable filesystems in the right places for Kubernetes to work. Immutably is good for very little if you can't run anything on it because the filesystem is read only.

I don't think it's as simple as "don't use that".

Not every Linux distribution is the best choice for every task. Pick the one that works with what you want to do.

[craigcabrey]

What do you mean by "working directory". Kubernetes (and k3s) need writable filesystems in several places in the host, and at least one of those needs to support overlayfs.

/var is the vast majority of disk usage of k8s (aside from workload specific things like a hostPath mount). That's still read-write. It's only / that is read-only in this new setup.

Anyways, there is a workaround, I just figured others in this community running CoreOS would find this useful.

[brandond]

Anyways, there is a workaround

Other than disabling composefs, what is it?

Other than /var/lib/rancher and /var/lib/kubelet, k3s will also want /etc/rancher writable, and /usr/local/bin if you're using the install script. It'll also want tmpfs or something similar at /tmp and /run.