Module migration support (#287)

README.md

@@ -2,29 +2,36 @@
 
 [![CircleCI](https://dl.circleci.com/status-badge/img/gh/dominodatalab/terraform-aws-eks/tree/main.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/dominodatalab/terraform-aws-eks/tree/main)
 
+:warning: Important: If you are currently using a version of the module >= `v3.0.0`  and want to update to >= `v3.5.0` you should take a look at the `bin/module-update/` utility.
+
 :warning: Important: If you have existing infrastructure created with a version of this module < `v3.0.0` you will need to migrate your state before updating the module to versions >= `v3.0.0`. See [state-migration](./bin/state-migration/README.md#terraform-state-migration-guide) for more details.
 
 :warning: Important: Starting from version v2.0.0, this module has KMS enabled by default. If you utilized a prior version without setting the kms.enabled variable, ensure you define kms.enabled = false to preserve your existing state. Failing to do so will default to kms.enabled = true, potentially causing the destruction of your existing infrastructure and possible data loss.
 
 
 ## Repository Structure
 
-* examples/:
+* `examples/`:
   * Purpose: Acts as an intuitive guide for module users.
   * Contents: Features end-user Terraform configurations along with illustrative `tfvars` samples, showcasing potential setups.
 
-* modules/:
+* `modules/`:
   * Purpose: Houses the primary modules that orchestrate the provisioning process.
   * Contents: These modules act as the main entry points for the setup of the underlying infrastructure beneath the **EKS** cluster and its associated components.
 
-* tests/:
+* `tests/`:
   * Purpose: Ensures the integrity and functionality of the module.
   * Contents: Contains automation-driven tests intended for validation and continuous integration (CI) checks.
 
-* bin/state-migration/:
+* `bin/state-migration/`:
   * Purpose: Contains automation to perform terraform state migration, from a monolithic module to a multi-module structure.
   * Contents: Script and documentation to perform terraform state migration.
 
+* `bin/module-update/`:
+  * A helper script to update the module version. Compatible only with modules on version `>= v3.0.0`.
+  * Contents: The directory includes the script and associated documentation for performing module updates.
+
+
 Always refer to each section's respective README or documentation for detailed information and usage guidelines.
 
 ## Prerequisites

bin/module-update/README.md

@@ -0,0 +1,182 @@
+# Module Migration Script for terraform-aws-eks
+
+This(`module-update.sh`) script automates the process of updating module versions for infrastructure and EKS clusters. It handles backups, updates to tfvars, and necessary imports and deprecations. It was tested updating from version `v3.0.5` to `v3.22.0`.
+
+## Prerequisites
+
+* Ensure the script is placed in the parent directory of your deployment folder.
+* Verify  [hcledit](https://github.com/minamijoyo/hcledit?tab=readme-ov-file#install), [tfvar](https://github.com/shihanng/tfvar?tab=readme-ov-file#installation), [aws-cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html#getting-started-install-instructions) and [terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli#install-terraform) are installed.
+
+* **:warning: IMPORTANT: Make sure to apply all changes in the current configuration to eliminate any drift not related to the module update.**
+
+
+## Steps
+
+1. Copy the script to the parent folder of your deployment folder.
+
+  ```shell
+  tree .
+  .
+  ├── domino-deploy-dir
+  └── module-update.sh
+  ```
+
+1. Set environment variables `DEPLOY_DIR`, `AWS_REGION` and `MOD_VERSION`:
+   1. `DEPLOY_DIR`: Deployment directory name.
+   2. `AWS_REGION`: Region where the module is deployed.
+   3. `MOD_VERSION`: the version of the module to upgrade to.
+
+      ```shell
+      export DEPLOY_DIR="domino-deploy-dir"
+      export AWS_REGION="us-east-1"
+      export MOD_VERSION="v3.22.0"
+      ```
+
+2. The script will perform the following steps when passing the `update` argument:
+   1. Download the new version of the module onto a a new directory which will be later deleted and set the module version to `MOD_VERSION`.
+   2. Create a backup copy of the current directory(i.e `domino-deploy-dir-BACKUP-24-10-21`)
+   3. Copy the `*.tf` and `*.sh` from the new module onto the existing directory.
+   4. Update the `*.tfvars` files with the defaults from the module while preserving existing values.
+   5. Attempts to perform corrections needed.
+   6. Delete the directory used for the initialization of the new module.
+3. Manual actions required.
+   1. if your EKS cluster has the `vpc-cni` addon installed, you need to edit `${DEPLOY_DIR}/terraform/cluster.tfvars` and to the list of addons. For example:
+   ```hcl
+    cluster_addons     = ["coredns", "kube-proxy", "vpc-cni"]
+   ```
+
+4. Run the Migration script.
+
+```shell
+./module-update.sh update
+```
+5. Move to the deployment directory and initialize components.
+```shell
+cd "$DEPLOY_DIR"
+./tf.sh all init
+```
+6. Plan, **Review Plan** and Apply each component.
+
+### Infra
+
+`./tf.sh infra plan`
+
+   * :warning: REVIEW `infra` plan
+
+  **Whats expected**
+
+  Expected Destroys:
+  ```shell
+  module.infra.module.storage.aws_s3_bucket_ownership_controls.monitoring will be destroyed
+  module.infra.module.storage.aws_s3_bucket_acl.monitoring will be destroyed
+  ```
+
+
+  **if MOD_VERSION >= https://github.com/dominodatalab/terraform-aws-eks/releases/tag/v3.6.0**
+
+```shell
+# │ Warning: Some objects will no longer be managed by Terraform
+# │
+# │ If you apply this plan, Terraform will discard its tracking information for the following objects, but it will not delete them:
+# │  - module.infra.aws_iam_policy.route53[0]
+
+```
+
+### :warning: IMPORTANT: Examine plan for unexpected destroys
+
+* Once the plan has been reviewed and deemed safe proceed to apply:
+
+`./tf.sh infra apply`
+
+
+### Cluster
+
+`./tf.sh cluster plan`
+
+   * :warning: REVIEW `cluster` plan
+  **Whats expected**
+
+  Expected Destroys:
+
+  ```
+   # module.eks.aws_eks_addon.vpc_cni will no longer be managed by Terraform, but will not be destroyed
+   # module.eks.module.k8s_setup[0].local_file.templates["eni_config"] will be destroyed
+  ```
+
+  **if MOD_VERSION >= https://github.com/dominodatalab/terraform-aws-eks/releases/tag/v3.5.0**
+  ```shell
+  # │ Warning: Some objects will no longer be managed by Terraform
+  # │
+  # │ If you apply this plan, Terraform will discard its tracking information for the following objects, but it will not delete them:
+  # │  - module.eks.aws_eks_addon.vpc_cni
+  ```
+### :warning: IMPORTANT: Examine plan for unexpected destroys
+
+* Once the plan has been reviewed and deemed safe proceed to apply:
+
+`./tf.sh cluster apply`
+
+
+### Nodes
+
+`./tf.sh nodes plan`
+
+If you see this error:
+
+```shell
+Error: Configuration for import target does not exist
+│
+│ The configuration for the given import module.nodes.aws_eks_addon.pre_compute_addons["vpc-cni"] does not exist. All target instances must have an associated configuration to be imported.
+╵
+```
+
+It is because `vpc-cni` is not in the list of EKS addons but there is an import for it.
+Add the `vpc-cni` to the list of EKS addons, example(you may have different addons, just make sure `vpc-cni` is in the list):
+```hcl
+eks = {
+  cluster_addons     = ["coredns", "kube-proxy","vpc-cni"]
+}
+```
+After the `terraform/cluster.tfvars` has been updated run `./tf.sh cluster apply` to update the outputs which are leveraged by the `nodes` module.
+
+:warning: In the unlikely scenario that you do not want to delete the `vpc-cni` addon remove the `terraform/nodes/imports.tf` file.
+
+
+
+* :warning: REVIEW `nodes` plan
+
+  **Whats expected**
+
+**if MOD_VERSION >= https://github.com/dominodatalab/terraform-aws-eks/releases/tag/v3.5.0**
+```shell
+# module.nodes.aws_eks_addon.pre_compute_addons["vpc-cni"] will be updated in-place
+# (imported from "mhekstest1:vpc-cni")
+```
+
+### :warning: IMPORTANT: Examine plan for unexpected destroys
+
+* Once the plan has been reviewed and deemed safe proceed to apply:
+`./tf.sh nodes apply`
+
+## Changes
+
+* The configuration behind `route53_hosted_zone_name` has been deprecated in favor or IRSA.
+* The `vpc-cni` EKS addon configuration has been moved onto the `nodes` module.
+
+
+## Manual action required
+
+1. if you were using the `route53_hosted_zone_name`, by default the module will not delete the IAM role and policy attached to the nodes, but in order to complete the setup the following annotation needs to be set in the `external-dns` service account. You can achieve this by adding the following release_overrides in the `domino.yaml`.
+    ```yaml
+    release_overrides:
+      external-dns:
+        chart_values:
+          serviceAccount:
+            annotations:
+              eks.amazonaws.com/role-arn: <arn for role named ‘<deploy_id>-external-dns’>
+              eks.amazonaws.com/sts-regional-endpoints: 'true'
+    ```
+   1. The value of the role(`eks.amazonaws.com/role-arn:`) for the `external_dns` service account can be obtained from the `cluster` output.
+      ```bash
+        ./tf.sh cluster output external_dns_irsa_role_arn
+      ```