Deploy CHT Core on Medic hosted EKS
Setting up a cloud hosted deployment of CHT Core on Medic’s AWS EKS infrastructure
While not directly available to the public who might be doing CHT Core development, having Medic’s process for using our Amazon Elastic Kubernetes Service (AWS EKS) publicly documented will help Medic employees new to EKS. As well, hopefully external developers looking to re-use Medic tools and process to use EKS will find it helpful.
While these instructions assume you work at Medic and have access to private GitHub repositories, many of the tools are fully open source.
Prerequisites
Command Line
Be sure you have these tools installed and repos cloned:
- awscli: version
2 or newer - kubectl: Must be within one minor version of cluster. If cluster is
1.24.x, use 1.23.x, 1.24.x or 1.25.x. - helm
- jq
- Medic Infra repo cloned
Optional: Autocomplete
Both helm and kubectl have autocomplete libraries. For power users and beginners alike, it adds a lot of discoverability. This code is for zsh, but bash, fish and powershell are supported as well:
source <(kubectl completion "$(basename "$SHELL")")
source <(helm completion "$(basename "$SHELL")")
See helm and kubectl docs to automatically loading these on every new session.
Request permission
By default, Medic teammates do not have EKS access and must file a ticket to request it:
- Create a ticket to get your DNS and Namespace created for EKS, which should match each other. As an example, a
mrjones-dev name space would match mrjones.dev.medicmobile.org DNS. The ticket should include requesting EKS access to be granted. - Once the ticket in step one is complete, follow the CLI setup guide.
NB - Security key (e.g. Yubikey) users need to add a TOTP MFA (Time-based, One-Time Password Multi-Factor Authentication) too! CLI requires the TOTP values (6-digit number) and security keys are not supported. Security keys can only be used on web logins.
First time setup
These steps only need to be run once!
After you have created a ticket per “Request permission” above, you should get a link to sign up for AWS. Click the link and:
Create new password ensure it’s 10+ characters including one alpha (a-z) and one special (~!@#$%^&*_-+=`|\(){}[]:;"'<>,.?/) character.
Setup MFA. In top-right corner of browser, there is a drop-down menu with your username @ medic. Click that and then on “My Security Credentials”
Assign an MFA device and give it the same name as your username: In AWS web GUI, click your name in upper right:
- Security Credentials
- scroll down to “Multi-factor authentication (MFA)”
- click “Assign MFA device”
- enter a “Device name” (should match username)
- “Select MFA device” that you’re using
Create Access Keys for Command Line Interface: In AWS web GUI, click your name in upper right -> Security Credentials -> scroll down to “Access keys” -> click “Create access key” -> for use case choose “Command Line Interface” -> click “Next” -> enter description and click “Create access key”
Run aws configure and place appropriate access keys during prompts. Use eu-west-2 region. It should look like this:
$ aws configure
@@ -380,12 +380,15 @@
"Value": "dev-cht-eks"
}
]
-
Switch to the production cluster and then find the subPath of the deployment you made the snapshot from. The COUCH-DB-NAME is usually cht-couchdb. But, it can sometimes be cht-couchdb-1 (check ./troubleshooting/list-deployments <your-namespace> if you still don’t know). Including the use-context, the two calls are:
kubectl config use-context arn:aws:eks:eu-west-2:720541322708:cluster/prod-cht-eks
-./troubleshooting/get-volume-binding <DEPLOYMENT> <COUCH-DB-NAME> | jq '.subPath'`
+
Switch to the production cluster and then find the subPath of the deployment you made the snapshot from. The COUCH-DB-NAME is usually cht-couchdb. But, it can sometimes be cht-couchdb-1 (check ./troubleshooting/list-deployments <your-namespace> if you still don’t know). Including the use-context, the two calls are below. Note that troubleshooting directory is in the CHT Core repo:
kubectl config use-context arn:aws:eks:eu-west-2:720541322708:cluster/prod-cht-eks
+./troubleshooting/get-volume-binding <DEPLOYMENT> <COUCH-DB-NAME> | jq '.subPath'
Which shows the path like this:
"storage/medic-core/couchdb/data"
-
Create a values.yml file from this template and edit the following fields:
project_name - likely your username followed by -dev. For example mrjones-devnamespace - likely the same as project name, your user followed by -dev. For example mrjones-devchtversion - this should match the version you cloned frompassword - this should match the version you cloned fromsecret - this should match the version you cloned fromuser - use medic useruuid - this should match the version you cloned fromcouchdb_node_storage_size - use the same size as the volume you just clonedhost - this should be your username followed by dev.medicmobile.org. For example mrjones.dev.medicmobile.orghosted_zone_id - this should always be Z3304WUAJTCM7PpreExistingDataAvailable - set this to be truepreExistingEBSVolumeID-1 - set this to be the ID from step 2. For example vol-f9dsa0f9sad09f0dsapreExistingEBSVolumeSize - use the same size as the volume you just cloneddataPathOnDiskForCouchDB - use the subPath you got in the step above. For example storage/medic-core/couchdb/data
Deploy this to development per the steps above. NB - Be sure to call kubectl config use-context arn:aws:eks:eu-west-2:720541322708:cluster/dev-cht-eks before you call ./cht-deploy! Always create test instances on the dev cluster.
Login using the user and password set above, which should match the production instance.
References and Debugging
More information on cht-deploy script is available in the CHT Core GitHub repository which includes specifics of the values.yaml file and more details about the debugging utilities listed below.
Debugging
A summary of the utilities in cht-core/scripts/deploy directory, assuming mrjones-dev namespace:
- list all resources:
./troubleshooting/list-all-resources mrjones-dev - view logs, assuming
cht-couchdb-1 returned from prior command: ./troubleshooting/view-logs mrjones-dev cht-couchdb-1 - describe deployment, assuming
cht-couchdb-1 returned from 1st command: ./troubleshooting/describe-deployment mrjones-dev cht-couchdb-1 - list all deployments:
./troubleshooting/list-all-resources mrjones-dev
Getting shell
Sometimes you need to look at files and other key pieces of data that are not available with the current troubleshooting/view-logs script. In this case, getting an interactive shell on the pod can be helpful.
- First, get a list pods for your namespace:
kubectl -n NAMESPACE get pods - After finding the pod you’re interested, connect to the pod to get a shell:
kubectl -n NAMESPACE exec -it PODNAME/CONTAINERNAME -- /bin/bash
invalid apiVersion Error
If you get the error:
exec plugin: invalid apiVersion “client.authentication.k8s.io/v1alpha1” when running kubectl version
You might be using an version of kubernetes api client.authentication.k8s.io which is not supported by your kubectl client. This can sometimes happen in EKS clusters if aws cli is an older version, in most cases you need at least version 2 of aws cli. Check version by running: aws --version and note that version 2 cannot be installed through pip (See Command Line section above for installation instructions)
SRE Steps for granting users access to a namespace
If you’re on the SRE/Infra team and want to grant a Medic teammate access to EKS:
- Tools required: aws, eksctl, kubectl
- Create AWS User.
- Attach IAM policy: Force_MFA and share auto-generated password safely
- Have user log in and finish MFA, access key setup
- SRE adds you to mfa-required-users group
- Add the namespaces and users to
tf/eks/dev/access/main.tf - Run tofu apply in the folder
tf/eks/dev/access - Create
identitymapping if needed:
Reading the AWS guide for principal access may help here!