k3s restore from backup fails when node IP changes

[ostapkostyk]

Hello,

I have a working multicloud HA cluster that uses external IP for flannel communication (initiated with --node-external-ip option).

Now I'd like to use tailscale vpn instead. Let's say to be able to add new master and agent nodes behind NAT.

So I did following:

• made an ETCD backup, stored it safely
• uninstalled k3s on all nodes
• installed a new k3s master with tailscale IP as node IP and public IP as external IP:
  curl -sfL https://get.k3s.io | sh -s - server --cluster-init --node-name=$NODE_NAME --token=$K3S_TOKEN --node-ip=$NODE_IP --node-external-ip=$EXT_IP --flannel-backend=wireguard-native --flannel-iface=tailscale0 --secrets-encryption --disable=traefik
• installed 2 more master nodes and joined the first server without issues. From this point, the plan was to restore ETCD from the backup.
• stopped all nodes.

  Note: if I try to follow documentation and apply k3s server --cluster-reset --cluster-reset-restore-path=PATH_TO_BACKUP as the first step without stopping "main" node, then the operation fails with the error message
  FATA[0000] starting kubernetes: preparing server: init cluster datastore and https: listen tcp :6443: bind: address already in use.

• executed
  k3s server --cluster-reset --cluster-reset-restore-path=PATH_TO_BACKUP. It fails with err msg
  level=fatal msg="/var/lib/rancher/k3s/server/cred/encryption-config.json, /var/lib/rancher/k3s/server/cred/encryption-state.json newer than datastore and> Apr 29 11:02:23 m1 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
• I deleted two above mentioned files and ran the same command again k3s server --cluster-reset --cluster-reset-restore-path=PATH_TO_BACKUP. Now it finished with the following error:

  level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [m1- 
  6b021274=https://PUBLIC_IP:2380], expect: m1-6b021274=https://TAILSCALE_IP:2380"
  Apr 29 11:28:16 m1 k3s[378680]: time="2024-04-29T11:28:16Z" level=warning msg="Failed to list nodes with etcd role: runtime core 
  not ready"
  Apr 29 11:28:16 m1 k3s[378680]: time="2024-04-29T11:28:16Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is 
  not ready: https://127.0.0.1:6443/v1-k3s/readyz: 500 Internal Server Error"
  

So, obviously the k3s expects to restore to the node with the same IP.
But how to restore k3s if I need to restore to different cluster? Or how to change a network of the cluster?

Regarding changing the network. This is actually my use-case at the moment. I'd like to use tailscale vpn instead of native mesh through public IPs.
If I just set --node-ip option, the cluster won't start. If I set this option only for the agent, it works but there is MTU mismatch which leads to severe issues. And seems that k3s sets MTU only when initializing the cluster for the first time (--cluster-init). That's why I tried to build a new cluster and than to restore it's state from ETCD backup.

[brandond]

So, obviously the k3s expects to restore to the node with the same IP.

No, that is not correct. The cluster-reset that is performed as part of the snapshot restore updates the cluster membership to the current node's IP.

Found [m1-6b021274=https://PUBLIC_IP:2380], expect: m1-6b021274=https://TAILSCALE_IP:2380"

You have specified --node-ip when running k3s server but not when running k3s server --cluster-reset. You need to consistently set your node IPs. This may be easier to do if you place it in the config.yaml, so you don't forget to set it on the command-line when running through the restore.

[ostapkostyk]

You are right, specifying --node-ip together with --cluster-reset solves the issue.

Thanks a lot!