Unschedulable Control Plane Nodes.

[mjordan79]

Summary

I use Microk8s in my company for a mid-sized cluster used for development and testing purposes. The cluster is configured with 3 master nodes (control planes) and 4 worker nodes. After the setup, I see the master nodes are schedulable and they will schedule applications. I think that a configuration for making control plane nodes unschedulable would be highly beneficial to a more robust HA configuration, to the point this can be a default setup for master nodes.

Why is this important?

On-premise installations might have the problem of limited resources and controlling them is crucial. In the eventuality of memory pressure for some of the nodes, master nodes can become unreliable and unresponsive, degrading the stability of the cluster.
Making control plane nodes free from scheduling applications, they can reliably continue to do what they're best at: controlling the cluster.

Is documentation also an alternative?

Yes. Considering that making a control plane node unschedulable would not be a "feature" of Microk8s per se, it could be beneficial to have some documentation that explain how to do it reliably and in a correct way. Which taints? Do the extension have the right tolerations to ignore master nodes? Is this an operation that should be done before installing extensions or after? Better to do this in individual nodes before forming the cluster?

[ktsakalozos]

Hi @mjordan79 thank you for pointing this out. You are right that in order to have a more stable cluster you would better have dedicated nodes to host the control plane. The reason why we do not enforce this policy out of the box is because we expect MicroK8s to be used to small and mid sized clusters. In such clusters we may not have the luxury of dedicated nodes.

Try setting the PreNoSchedule taint to the control plane nodes of a MicroK8s cluster with:

microk8s.kubectl taint nodes --selector=node.kubernetes.io/microk8s-controlplane=microk8s-controlplane cp-node=true:PreferNoSchedule

From: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[utamas]

Oh, cool. It seems to be working.