Accidentally removed k3s (init) master node

[ffly90]

I accidentally deleted one of three master nodes in my k3s cluster. The node I deleted was originally the init node for the cluster. Is there a procedure to rejoin the node to the cluster and how i would do this? Etcd snapshots are available to perform a disaster recovery, we would however prefer to get it back to normal without a downtime/cluster-reset.

Current state of the cluster:

$ kubectl get nodes
NAME         STATUS   ROLES                       AGE    VERSION
server2   Ready    control-plane,etcd,master   499d   v1.25.6+k3s1
server3   Ready    control-plane,etcd,master   499d   v1.25.6+k3s1
worker1  Ready    agent                       394d   v1.25.6+k3s1
worker2  Ready    agent                       394d   v1.25.6+k3s1
worker3  Ready    agent                       394d   v1.25.6+k3s1

[brandond]

There is nothing special about that first node once you have other members in the cluster. Just change the --cluster-init flag to --server=server2, delete the database directory from disk, restart k3s, and it should join back up.

[ffly90]

Thank you @brandond, we did not know that there is nothing special about the init node after multiple server nodes have joined the cluster. With this knowledge and the removal of the database folder, we could successfully rejoin the lost server.

[benbonnet]

am getting started, trying to get a concise idea of what is happening when removing a master node.

So far i'm first draining the node (kubectl drain xx), then delete the node (kubectl delete node xx). My issue is, k3s invariably crashes. Look like etcd keeps trying to find the deleted node (in the logs, failed to publish local member to cluster through raft). Moreover, systemctl restart k3s just hangs forever.

I'd be looking to get a robust process to add/remove master nodes; mainly to feel safe about such issue. Would yo have any recommendations ? should I manually manipulate etcd to remove the deleted node ref ?

In case you could provide an example of the full command typed to reinitialize the cluster (am not finding any accurate doc about flagging -cluster-init with --server=xx2 you mentionning); that be just great!