Update workload cluster restore instruction

docs/content/en/docs/clustermgmt/cluster-backup-restore/restore-cluster.md

@@ -161,4 +161,177 @@ If the cluster is no longer accessible in any means, or the infrastructure machi
 
 ## Restore a workload cluster
 
-Restoring a workload cluster is a delicate process. If you have an [EKS Anywhere Enterprise Subscription](https://aws.amazon.com/eks/eks-anywhere/pricing/), please contact AWS support team if you wish to perform such operation.
+### Cluster accessible and the infrastructure components not changed after etcd backup was taken
+
+Similar to the failed management cluster without infrastructure components change situation, follow the [External etcd backup and restore]({{< relref "../etcd-backup-restore/etcdbackup" >}}) to restore the workload cluster itself from the backup.
+
+### Cluster not accessible or infrastructure components changed after etcd backup was taken
+
+If the workload cluster is still accessible, but the infrastructure machines are changed after the etcd backup was taken, you can still try restoring the cluster itself from the etcd backup. Although doing so is risky: it can potentially cause the node names, IPs and other infrastructure configurations to revert to a state that is no longer valid. Restoring etcd effectively takes a cluster back in time and all clients will experience a conflicting, parallel history. This can impact the behavior of watching components like Kubernetes controller managers, EKS Anywhere cluster controller manager, and Cluster API controller managers.
+
+If the original workload cluster becomes inaccessible or cannot be restored to a healthy state from an outdated etcd, a new workload cluster needs to be created. This new cluster should be managed by the same management cluster that oversaw the original. You must then restore your workload applications to this new cluster from the etcd backup of the original. This ensures the management cluster retains control, with all data from the old cluster intact. Below is an example of applying the etcd `backup etcd-snapshot-w01.db` from a failed workload cluster `w01` to a new cluster `w02`:
+
+1. Create a new workload cluster to which you will be migrating your workloads and applications from the original failed workload cluster.
+
+    You can define a cluster config similar to your old workload cluster, with a different cluster name (if the old workload cluster still exists), and run cluster creation of the new workload cluster with the **exact same EKS Anywhere version** used to create the old workload cluster.
+
+    ```bash
+    export MGMT_CLUSTER="mgmt"
+    export MGMT_CLUSTER_KUBECONFIG=${MGMT_CLUSTER}/${MGMT_CLUSTER}-eks-a-cluster.kubeconfig
+
+    eksctl anywhere create cluster -f w02.yaml --kubeconfig $MGMT_CLUSTER_KUBECONFIG
+    ```
+
+1. Save the config map objects of the new workload cluster to a file.
+
+    Save a copy of the new workload cluster's `cluster-info`, `kube-proxy` and `kubeadm-config` config map objects before the restore. This is necessary as the etcd restore will override the config maps above with the metadata information (certificates, endpoint, etc.) from the old cluster.
+
+    ```bash
+    export WORKLOAD_CLUSTER_NAME="w02"
+    export WORKLOAD_CLUSTER_KUBECONFIG=${WORKLOAD_CLUSTER_NAME}/${WORKLOAD_CLUSTER_NAME}-eks-a-cluster.kubeconfig
+
+    cat <<EOF >> w02-cm.yaml
+    $(kubectl get -n kube-public cm cluster-info -oyaml --kubeconfig $WORKLOAD_CLUSTER_KUBECONFIG)
+    ---
+    $(kubectl get -n kube-system cm kube-proxy -oyaml --kubeconfig $WORKLOAD_CLUSTER_KUBECONFIG)
+    ---
+    $(kubectl get -n kube-system cm kubeadm-config -oyaml --kubeconfig $WORKLOAD_CLUSTER_KUBECONFIG)
+    EOF
+    ```
+
+    Manually remove the `creationTimestamp`, `resourceVersion`, `uid` from the config map objects, so that later you can run `kubectl apply` against this file without errors.
+
+
+1. Follow the [External etcd backup and restore]({{< relref "../etcd-backup-restore/etcdbackup" >}}) to restore the old workload cluster's etcd backup `etcd-snapshot-w01.db` onto the new workload cluster `w02`. Use different restore process based on OS family:
+    * [BottleRocket]({{< relref "../etcd-backup-restore/bottlerocket-etcd-backup/#restore-etcd-from-backup" >}})
+    * [Ubuntu]({{< relref "../etcd-backup-restore/ubuntu-rhel-etcd-backup/#restore" >}})
+
+    {{% alert title="Warning" color="warning" %}}
+
+    Do not unpause the cluster reconcilers as the workload cluster is not completely setup with required configurations yet. Unpausing the cluster reconcilers might cause unexpected machine rollout and leave the cluster in unhealthy state.
+
+    {{% /alert %}}
+
+    You might notice that after restoring the original etcd backup to the new workload cluster `w02`, all the nodes go to `NotReady` state with node names changed to have prefix `w01-*`. This is because restoring etcd effectively applies the node data from the original cluster which causes a conflicting history and can impact the behavior of watching components like Kubelets, Kubernetes controller managers.
+
+    ```bash
+    kubectl get nodes --kubeconfig $WORKLOAD_CLUSTER_KUBECONFIG
+
+    NAME                              STATUS     ROLES           AGE     VERSION
+    w01-bbtdd                         NotReady   control-plane   3d23h   v1.27.3-eks-6f07bbc
+    w01-md-0-66dbcfb56cxng8lc-8ppv5   NotReady   <none>          3d23h   v1.27.3-eks-6f07bbc
+    ```
+
+1. Restart Kubelet of the control plane and worker nodes of the new workload cluster after the restore.
+
+    For some cases, Kubelet on the node will automatically restart and nodes becomes ready. For other cases, you need to manually restart the Kubelet on all the control plane and worker nodes in order to bring back the nodes to ready state. Kubelet registers the node itself with the apisever which then updates etcd with the correct node data of the new workload cluster `w02`.
+
+    {{< tabpane >}}
+    {{< tab header="Ubuntu or RHEL" lang="bash" >}}
+# SSH into the control plane and worker nodes. You must do this for each node.
+ssh -i ${SSH_KEY} ${SSH_USERNAME}@<node IP>
+sudo su
+systemctl restart kubelet
+    {{< /tab >}}
+    {{< tab header="Bottlerocket" lang="bash" >}}
+# SSH into the control plane and worker nodes. You must do this for each node.
+ssh -i ${SSH_KEY} ${SSH_USERNAME}@<node IP>
+apiclient exec admin bash
+sheltie
+systemctl restart kubelet
+    {{< /tab >}}
+    {{< /tabpane >}}
+
+1. Add back `node-role.kubernetes.io/control-plane` label to **all** the control plane nodes.
+
+    ```bash
+    kubectl label nodes <control-plane-node-name> node-role.kubernetes.io/control-plane= --kubeconfig $WORKLOAD_CLUSTER_KUBECONFIG
+    ```
+
+1. Remove the lagacy nodes (if any).
+
+    ```bash
+    kubectl get nodes --kubeconfig $WORKLOAD_CLUSTER_KUBECONFIG
+
+    NAME                              STATUS     ROLES           AGE     VERSION
+    w01-bbtdd                         NotReady   control-plane   3d23h   v1.27.3-eks-6f07bbc
+    w01-md-0-66dbcfb56cxng8lc-8ppv5   NotReady   <none>          3d23h   v1.27.3-eks-6f07bbc
+    w02-fcbm2j                        Ready      control-plane   91m     v1.27.3-eks-6f07bbc
+    w02-md-0-b7cc67cd4xd86jf-4c9ktp   Ready      <none>          73m     v1.27.3-eks-6f07bbc
+    ```
+
+    ```bash
+    kubectl delete node w01-bbtdd --kubeconfig $WORKLOAD_CLUSTER_KUBECONFIG
+    kubectl delete node w01-md-0-66dbcfb56cxng8lc-8ppv5 --kubeconfig $WORKLOAD_CLUSTER_KUBECONFIG
+    ```
+
+1. Re-apply the original config map objects of the workload cluster.
+
+    Re-apply the `cluster-info`, `kube-proxy` and `kubeadm-config` config map objects we saved in previous step to the workload cluster `w02`.
+
+    ```bash
+    kubectl apply -f w02-cm.yaml --kubeconfig $WORKLOAD_CLUSTER_KUBECONFIG
+    ```
+
+1. Validate the nodes are in ready state.
+
+    ```bash
+    kubectl get nodes --kubeconfig $WORKLOAD_CLUSTER_KUBECONFIG
+
+    NAME                              STATUS   ROLES           AGE     VERSION
+    w02-djshz                         Ready    control-plane   9m7s    v1.27.3-eks-6f07bbc
+    w02-md-0-6bbc8dd6d4xbgcjh-wfmb6   Ready    <none>          3m55s   v1.27.3-eks-6f07bbc
+    ```
+
+1. Restart the system pods to ensure that they use the config maps you re-applied in previous step.
+
+    ```bash
+    kubectl rollout restart ds kube-proxy -n kube-system --kubeconfig $WORKLOAD_CLUSTER_KUBECONFIG
+    ```
+
+1. Unpause the cluster reconcilers
+
+    ```bash
+    kubectl annotate clusters.anywhere.eks.amazonaws.com $WORKLOAD_CLUSTER_NAME anywhere.eks.amazonaws.com/paused- --kubeconfig=$MGMT_CLUSTER_KUBECONFIG
+
+    kubectl patch clusters.cluster.x-k8s.io $WORKLOAD_CLUSTER_NAME --type merge -p '{"spec":{"paused": false}}' -n eksa-system --kubeconfig=$MGMT_CLUSTER_KUBECONFIG
+    ```
+
+1. Rollout and restart all the machine objects so that the workload cluster has a clean state.
+
+    ```bash
+    # Substitute the EKS Anywhere release version with whatever CLI version you are using
+    EKSA_RELEASE_VERSION=v0.18.3
+    BUNDLE_MANIFEST_URL=$(curl -s https://anywhere-assets.eks.amazonaws.com/releases/eks-a/manifest.yaml | yq ".spec.releases[] | select(.version==\"$EKSA_RELEASE_VERSION\").bundleManifestUrl")
+    CLI_TOOLS_IMAGE=$(curl -s $BUNDLE_MANIFEST_URL | yq ".spec.versionsBundles[0].eksa.cliTools.uri")
+
+
+    # Rollout restart all the control plane machines
+    docker run -i --network host -w $(pwd) -v $(pwd):/$(pwd) --entrypoint clusterctl ${CLI_TOOLS_IMAGE} alpha rollout restart kubeadmcontrolplane/${WORKLOAD_CLUSTER_NAME} -n eksa-system --kubeconfig ${MGMT_CLUSTER_KUBECONFIG}
+
+    # Rollout restart all the worker machines
+    # You need to repeat below command for each worker node group
+    docker run -i --network host -w $(pwd) -v $(pwd):/$(pwd) --entrypoint clusterctl ${CLI_TOOLS_IMAGE} alpha rollout restart machinedeployment/${WORKLOAD_CLUSTER_NAME}-md-0 -n eksa-system --kubeconfig ${MGMT_CLUSTER_KUBECONFIG}
+    ```
+
+1. Validate the new workload cluster is in the desired state.
+
+    ```bash
+    kubectl get clusters -n default -o custom-columns="NAME:.metadata.name,READY:.status.conditions[?(@.type=='Ready')].status" --kubeconfig $MGMT_CLUSTER_KUBECONFIG
+
+    NAME   READY
+    mgmt   True
+    w02    True
+
+    kubectl get clusters.cluster.x-k8s.io -n eksa-system --kubeconfig $MGMT_CLUSTER_KUBECONFIG
+
+    NAME   PHASE         AGE
+    mgmt   Provisioned   11h   
+    w02    Provisioned   11h 
+
+    kubectl get kcp -n eksa-system  --kubeconfig $MGMT_CLUSTER_KUBECONFIG
+
+    NAME   CLUSTER    INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE   VERSION
+    mgmt   mgmt       true          true                   2          2       2                       11h   v1.27.1-eks-1-27-4
+    w02    w02        true          true                   2          2       2                       11h   v1.27.1-eks-1-27-4
+    ```

docs/content/en/docs/clustermgmt/cluster-rebootnode.md

@@ -42,17 +42,8 @@ If this cannot be verified, do not proceed any further
     - add the paused annotation to the EKSA clusters and CAPI clusters: 
     ```bash
     kubectl annotate clusters.anywhere.eks.amazonaws.com $CLUSTER_NAME anywhere.eks.amazonaws.com/paused=true --kubeconfig=$MGMT_KUBECONFIG
-    ```
 
-    **NOTE**: If you are using vSphere provider, it is also necessary to set `cluster.spec.paused` to true. For example:
-    ```bash
-    kubectl edit clusters.cluster.x-k8s.io -n eksa-system $CLUSTER_NAME --kubeconfig=$MGMT_KUBECONFIG
-    ```
-    add the `paused: true` line under the spec section:
-    ```bash
-    ...
-    spec:
-      paused: true
+    kubectl patch clusters.cluster.x-k8s.io $CLUSTER_NAME --type merge -p '{"spec":{"paused": true}}' -n eksa-system --kubeconfig=$MGMT_KUBECONFIG
     ```
 
 1. For all of the nodes in the cluster, perform the following steps in this order: worker nodes, control plane nodes, and etcd nodes.
@@ -83,15 +74,6 @@ If this cannot be verified, do not proceed any further
 1. Remove the paused annotations from EKS Anywhere cluster.
     ```bash
     kubectl annotate clusters.anywhere.eks.amazonaws.com $CLUSTER_NAME anywhere.eks.amazonaws.com/paused- --kubeconfig=$MGMT_KUBECONFIG
-    ```
 
-    **NOTE**: If you are using vSphere provider, it is also necessary to set `cluster.spec.paused` to false
-    ```bash
-    kubectl edit clusters.cluster.x-k8s.io -n eksa-system $CLUSTER_NAME --kubeconfig=$MGMT_KUBECONFIG
+    kubectl patch clusters.cluster.x-k8s.io $CLUSTER_NAME --type merge -p '{"spec":{"paused": false}}' -n eksa-system --kubeconfig=$MGMT_KUBECONFIG
     ```
-    set paused in the spec section to false:
-    ```bash
-    ...
-    spec:
-      paused: false
-    ```

docs/content/en/docs/clustermgmt/etcd-backup-restore/bottlerocket-etcd-backup.md

@@ -116,11 +116,9 @@ Make sure to setup the [admin environment variables]({{< relref "#admin-machine-
 
     Before starting the process of restoring etcd, you have to pause some cluster reconciliation objects so EKS Anywhere doesn't try to perform any operations on the cluster while you restore the etcd snapshot.
     ```bash
-    # Pause control plane reconcilation
-    kubectl --kubeconfig=${MANAGEMENT_KUBECONFIG} -n eksa-system annotate machinehealthchecks ${CLUSTER_NAME}-kcp-unhealthy cluster.x-k8s.io/paused=true
+    kubectl annotate clusters.anywhere.eks.amazonaws.com $CLUSTER_NAME anywhere.eks.amazonaws.com/paused=true --kubeconfig=$MANAGEMENT_KUBECONFIG
 
-    # Pause etcd reconcilation
-    kubectl --kubeconfig=${MANAGEMENT_KUBECONFIG} -n eksa-system annotate etcdadmclusters ${CLUSTER_NAME}-etcd cluster.x-k8s.io/paused=true
+    kubectl patch clusters.cluster.x-k8s.io $CLUSTER_NAME --type merge -p '{"spec":{"paused": true}}' -n eksa-system --kubeconfig=$MANAGEMENT_KUBECONFIG
     ```
 
 2. Stop control plane core components
@@ -189,32 +187,32 @@ Make sure to setup the [admin environment variables]({{< relref "#admin-machine-
     export ETCD_CONTAINER_ID=$(ctr -n k8s.io c ls | grep -w "etcd-io" | head -1 | cut -d " " -f1)
     ```
 
-    # run the restore command
-
     {{< tabpane >}}
 
     {{< tab header="Using etcd < v3.5.x" lang="bash" >}}
-    ctr -n k8s.io t exec -t --exec-id etcd ${ETCD_CONTAINER_ID} etcdctl \
-        snapshot restore /var/lib/etcd/data/etcd-snapshot.db \
-        --name=${ETCD_NAME} \
-        --initial-cluster=${ETCD_INITIAL_CLUSTER} \
-        --initial-cluster-token=${INITIAL_CLUSTER_TOKEN} \
-        --initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
-        --cacert=/var/lib/etcd/pki/ca.crt \
-        --cert=/var/lib/etcd/pki/server.crt \
-        --key=/var/lib/etcd/pki/server.key
+# run the restore command
+ctr -n k8s.io t exec -t --exec-id etcd ${ETCD_CONTAINER_ID} etcdctl \
+    snapshot restore /var/lib/etcd/data/etcd-snapshot.db \
+    --name=${ETCD_NAME} \
+    --initial-cluster=${ETCD_INITIAL_CLUSTER} \
+    --initial-cluster-token=${INITIAL_CLUSTER_TOKEN} \
+    --initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
+    --cacert=/var/lib/etcd/pki/ca.crt \
+    --cert=/var/lib/etcd/pki/server.crt \
+    --key=/var/lib/etcd/pki/server.key
     {{< /tab >}}
 
     {{< tab header="Using etcd >= v3.5.x" lang="bash" >}}
-    ctr -n k8s.io t exec -t --exec-id etcd ${ETCD_CONTAINER_ID} etcdutl \
-        snapshot restore /var/lib/etcd/data/etcd-snapshot.db \
-        --name=${ETCD_NAME} \
-        --initial-cluster=${ETCD_INITIAL_CLUSTER} \
-        --initial-cluster-token=${INITIAL_CLUSTER_TOKEN} \
-        --initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
-        --cacert=/var/lib/etcd/pki/ca.crt \
-        --cert=/var/lib/etcd/pki/server.crt \
-        --key=/var/lib/etcd/pki/server.key
+# run the restore command
+ctr -n k8s.io t exec -t --exec-id etcd ${ETCD_CONTAINER_ID} etcdutl \
+    snapshot restore /var/lib/etcd/data/etcd-snapshot.db \
+    --name=${ETCD_NAME} \
+    --initial-cluster=${ETCD_INITIAL_CLUSTER} \
+    --initial-cluster-token=${INITIAL_CLUSTER_TOKEN} \
+    --initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
+    --cacert=/var/lib/etcd/pki/ca.crt \
+    --cert=/var/lib/etcd/pki/server.crt \
+    --key=/var/lib/etcd/pki/server.key
     {{< /tab >}}
 
     {{< /tabpane >}} 
@@ -235,6 +233,9 @@ Make sure to setup the [admin environment variables]({{< relref "#admin-machine-
     # copy over the new etcd data files to the data directory
     mv /tmp/${ETCD_NAME}.etcd/member /var/lib/etcd/data/
 
+    # copy the WAL folder for the cluster member data before the restore to the data/member directory
+    cp -r /var/lib/etcd/data/member.backup/wal /var/lib/etcd/data/member
+
     # re-start the etcd pod
     mv /tmp/temp-manifests/* /etc/kubernetes/manifests/
     ```
@@ -289,12 +290,11 @@ Make sure to setup the [admin environment variables]({{< relref "#admin-machine-
 1. Unpause the cluster reconcilers
 
     Once the etcd restore is complete, you can resume the cluster reconcilers.
+
     ```bash
-    # unpause control plane reconcilation
-    kubectl --kubeconfig=${MANAGEMENT_KUBECONFIG} -n eksa-system annotate machinehealthchecks ${CLUSTER_NAME}-kcp-unhealthy cluster.x-k8s.io/paused-
+    kubectl annotate clusters.anywhere.eks.amazonaws.com $CLUSTER_NAME anywhere.eks.amazonaws.com/paused- --kubeconfig=$MANAGEMENT_KUBECONFIG
 
-    # unpause etcd reconcilation
-    kubectl --kubeconfig=${MANAGEMENT_KUBECONFIG} -n eksa-system annotate etcdadmclusters ${CLUSTER_NAME}-etcd cluster.x-k8s.io/paused-
+    kubectl patch clusters.cluster.x-k8s.io $CLUSTER_NAME --type merge -p '{"spec":{"paused": false}}' -n eksa-system --kubeconfig=$MANAGEMENT_KUBECONFIG
     ```
 
 At this point you should have the etcd cluster restored to snapshot.

docs/content/en/docs/clustermgmt/etcd-backup-restore/ubuntu-rhel-etcd-backup.md

@@ -55,18 +55,18 @@ scp -i $PRIV_KEY ec2-user@$ETCD_VM_IP:/home/ec2-user/snapshot.db .
 
 NOTE: This snapshot file contains all information stored in the cluster, so make sure you save it securely (encrypt it).
 
-
 ### Restore
 
 Restoring etcd is a 2-part process. The first part is restoring etcd using the snapshot, creating a new data-dir for etcd. The second part is replacing the current etcd data-dir with the one generated after restore. During etcd data-dir replacement, we cannot have any kube-apiserver instances running in the cluster. So we will first stop all instances of kube-apiserver and other controlplane components using the following steps for every controlplane VM:
 
 #### Pausing Etcdadm cluster and control plane machine health check reconcile
 
 During restore, it is required to pause the Etcdadm controller reconcile and the control plane machine healths checks for the target cluster (whether it is management or workload cluster). To do that, you need to add a `cluster.x-k8s.io/paused` annotation to the target cluster's `etcdadmclusters` and `machinehealthchecks` resources. For example,
+
 ```bash
-kubectl annotate machinehealthchecks ${CLUSTER_NAME}-kcp-unhealthy cluster.x-k8s.io/paused=true -n eksa-system --kubeconfig mgmt-cluster.kubeconfig
+kubectl annotate clusters.anywhere.eks.amazonaws.com $CLUSTER_NAME anywhere.eks.amazonaws.com/paused=true --kubeconfig mgmt-cluster.kubeconfig
 
-kubectl annotate etcdadmclusters ${CLUSTER_NAME}-etcd cluster.x-k8s.io/paused=true -n eksa-system --kubeconfig mgmt-cluster.kubeconfig
+kubectl patch clusters.cluster.x-k8s.io $CLUSTER_NAME --type merge -p '{"spec":{"paused": true}}' -n eksa-system --kubeconfig mgmt-cluster.kubeconfig
 ```
 
 #### Stopping the controlplane components
@@ -166,8 +166,9 @@ mv temp-manifests/*.yaml /etc/kubernetes/manifests
 #### Resuming Etcdadm cluster and control plane machine health checks reconcile
 
 Resume Etcdadm cluster reconcile and control plane machine health checks for the target cluster by removing the `cluster.x-k8s.io/paused` annotation in the target cluster's  resource. For example,
+
 ```bash
-kubectl annotate machinehealthchecks ${CLUSTER_NAME}-kcp-unhealthy cluster.x-k8s.io/paused- -n eksa-system --kubeconfig mgmt-cluster.kubeconfig
+kubectl annotate clusters.anywhere.eks.amazonaws.com $CLUSTER_NAME anywhere.eks.amazonaws.com/paused- --kubeconfig mgmt-cluster.kubeconfig
 
-kubectl annotate etcdadmclusters ${CLUSTER_NAME}-etcd cluster.x-k8s.io/paused- -n eksa-system --kubeconfig mgmt-cluster.kubeconfig
+kubectl patch clusters.cluster.x-k8s.io $CLUSTER_NAME --type merge -p '{"spec":{"paused": false}}' -n eksa-system --kubeconfig mgmt-cluster.kubeconfig
 ```