Skip to content

Commit

Permalink
PLAT-5932: Inject user data for node groups. (#11)
Browse files Browse the repository at this point in the history 
Generate user data for all node groups when an AMI is provided.
One can specify additional args.
  • Loading branch information
Michael Fraenkel authored Nov 21, 2022
1 parent 64b67c2 commit e44fc56
Show file tree
Hide file tree
Showing 9 changed files with 125 additions and 70 deletions.
4 changes: 2 additions & 2 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -109,12 +109,12 @@ aws s3 rb s3://"${AWS_TERRAFORM_REMOTE_STATE_BUCKET}" --force

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_additional_node_groups"></a> [additional\_node\_groups](#input\_additional\_node\_groups) | Additional EKS managed node groups definition. | <pre>map(object({<br> ami = optional(string)<br> instance_types = list(string)<br> spot = optional(bool, false)<br> min_per_az = number<br> max_per_az = number<br> desired_per_az = number<br> labels = map(string)<br> taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])<br> tags = optional(map(string), {})<br> volume = object({<br> size = string<br> type = string<br> })<br> }))</pre> | `{}` | no |
| <a name="input_additional_node_groups"></a> [additional\_node\_groups](#input\_additional\_node\_groups) | Additional EKS managed node groups definition. | <pre>map(object({<br> ami = optional(string)<br> bootstrap_extra_args = optional(string, "")<br> instance_types = list(string)<br> spot = optional(bool, false)<br> min_per_az = number<br> max_per_az = number<br> desired_per_az = number<br> labels = map(string)<br> taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])<br> tags = optional(map(string), {})<br> volume = object({<br> size = string<br> type = string<br> })<br> }))</pre> | `{}` | no |
| <a name="input_availability_zones"></a> [availability\_zones](#input\_availability\_zones) | List of Availibility zones to distribute the deployment, EKS needs at least 2,https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html.<br> Note that setting this variable bypasses validation of the status of the zones data 'aws\_availability\_zones' 'available'.<br> Caller is responsible for validating status of these zones. | `list(string)` | `[]` | no |
| <a name="input_bastion_ami_id"></a> [bastion\_ami\_id](#input\_bastion\_ami\_id) | AMI ID for the bastion EC2 instance, otherwise we will use the latest 'amazon\_linux\_2' ami | `string` | `""` | no |
| <a name="input_cidr"></a> [cidr](#input\_cidr) | The IPv4 CIDR block for the VPC. | `string` | `"10.0.0.0/16"` | no |
| <a name="input_create_bastion"></a> [create\_bastion](#input\_create\_bastion) | Create bastion toggle. | `bool` | `false` | no |
| <a name="input_default_node_groups"></a> [default\_node\_groups](#input\_default\_node\_groups) | EKS managed node groups definition. | <pre>object(<br> {<br> compute = object(<br> {<br> ami = optional(string)<br> instance_types = optional(list(string), ["m5.2xlarge"])<br> spot = optional(bool, false)<br> min_per_az = optional(number, 0)<br> max_per_az = optional(number, 10)<br> desired_per_az = optional(number, 1)<br> labels = optional(map(string), {<br> "dominodatalab.com/node-pool" = "default"<br> })<br> taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])<br> tags = optional(map(string), {})<br> volume = optional(object(<br> {<br> size = optional(number, 100)<br> type = optional(string, "gp3")<br> }),<br> {<br> size = 100<br> type = "gp3"<br> }<br> )<br> }),<br> platform = object(<br> {<br> ami = optional(string)<br> instance_types = optional(list(string), ["m5.4xlarge"])<br> spot = optional(bool, false)<br> min_per_az = optional(number, 1)<br> max_per_az = optional(number, 10)<br> desired_per_az = optional(number, 1)<br> labels = optional(map(string), {<br> "dominodatalab.com/node-pool" = "platform"<br> })<br> taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])<br> tags = optional(map(string), {})<br> volume = optional(object(<br> {<br> size = optional(number, 100)<br> type = optional(string, "gp3")<br> }),<br> {<br> size = 100<br> type = "gp3"<br> }<br> )<br> }),<br> gpu = object(<br> {<br> ami = optional(string)<br> instance_types = optional(list(string), ["g4dn.xlarge"])<br> spot = optional(bool, false)<br> min_per_az = optional(number, 0)<br> max_per_az = optional(number, 10)<br> desired_per_az = optional(number, 0)<br> labels = optional(map(string), {<br> "dominodatalab.com/node-pool" = "default-gpu"<br> "nvidia.com/gpu" = true<br> })<br> taints = optional(list(object({ key = string, value = optional(string), effect = string })), [<br> { key = "nvidia.com/gpu", value = "true", effect = "NO_SCHEDULE" }<br> ])<br> tags = optional(map(string), {})<br> volume = optional(object(<br> {<br> size = optional(number, 100)<br> type = optional(string, "gp3")<br> }),<br> {<br> size = 100<br> type = "gp3"<br> }<br> )<br> })<br> })</pre> | <pre>{<br> "compute": {},<br> "gpu": {},<br> "platform": {}<br>}</pre> | no |
| <a name="input_default_node_groups"></a> [default\_node\_groups](#input\_default\_node\_groups) | EKS managed node groups definition. | <pre>object(<br> {<br> compute = object(<br> {<br> ami = optional(string)<br> bootstrap_extra_args = optional(string, "")<br> instance_types = optional(list(string), ["m5.2xlarge"])<br> spot = optional(bool, false)<br> min_per_az = optional(number, 0)<br> max_per_az = optional(number, 10)<br> desired_per_az = optional(number, 1)<br> labels = optional(map(string), {<br> "dominodatalab.com/node-pool" = "default"<br> })<br> taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])<br> tags = optional(map(string), {})<br> volume = optional(object(<br> {<br> size = optional(number, 100)<br> type = optional(string, "gp3")<br> }),<br> {<br> size = 100<br> type = "gp3"<br> }<br> )<br> }),<br> platform = object(<br> {<br> ami = optional(string)<br> bootstrap_extra_args = optional(string, "")<br> instance_types = optional(list(string), ["m5.4xlarge"])<br> spot = optional(bool, false)<br> min_per_az = optional(number, 1)<br> max_per_az = optional(number, 10)<br> desired_per_az = optional(number, 1)<br> labels = optional(map(string), {<br> "dominodatalab.com/node-pool" = "platform"<br> })<br> taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])<br> tags = optional(map(string), {})<br> volume = optional(object(<br> {<br> size = optional(number, 100)<br> type = optional(string, "gp3")<br> }),<br> {<br> size = 100<br> type = "gp3"<br> }<br> )<br> }),<br> gpu = object(<br> {<br> ami = optional(string)<br> bootstrap_extra_args = optional(string, "")<br> instance_types = optional(list(string), ["g4dn.xlarge"])<br> spot = optional(bool, false)<br> min_per_az = optional(number, 0)<br> max_per_az = optional(number, 10)<br> desired_per_az = optional(number, 0)<br> labels = optional(map(string), {<br> "dominodatalab.com/node-pool" = "default-gpu"<br> "nvidia.com/gpu" = true<br> })<br> taints = optional(list(object({ key = string, value = optional(string), effect = string })), [<br> { key = "nvidia.com/gpu", value = "true", effect = "NO_SCHEDULE" }<br> ])<br> tags = optional(map(string), {})<br> volume = optional(object(<br> {<br> size = optional(number, 100)<br> type = optional(string, "gp3")<br> }),<br> {<br> size = 100<br> type = "gp3"<br> }<br> )<br> })<br> })</pre> | <pre>{<br> "compute": {},<br> "gpu": {},<br> "platform": {}<br>}</pre> | no |
| <a name="input_deploy_id"></a> [deploy\_id](#input\_deploy\_id) | Domino Deployment ID. | `string` | `"domino-eks"` | no |
| <a name="input_efs_access_point_path"></a> [efs\_access\_point\_path](#input\_efs\_access\_point\_path) | Filesystem path for efs. | `string` | `"/domino"` | no |
| <a name="input_eks_master_role_names"></a> [eks\_master\_role\_names](#input\_eks\_master\_role\_names) | IAM role names to be added as masters in eks. | `list(string)` | `[]` | no |
Expand Down
4 changes: 2 additions & 2 deletions submodules/eks/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -60,10 +60,10 @@ No modules.

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_additional_node_groups"></a> [additional\_node\_groups](#input\_additional\_node\_groups) | Additional EKS managed node groups definition. | <pre>map(object({<br> ami = optional(string)<br> instance_types = list(string)<br> spot = optional(bool, false)<br> min_per_az = number<br> max_per_az = number<br> desired_per_az = number<br> labels = map(string)<br> taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])<br> tags = optional(map(string), {})<br> volume = object({<br> size = string<br> type = string<br> })<br> }))</pre> | `{}` | no |
| <a name="input_additional_node_groups"></a> [additional\_node\_groups](#input\_additional\_node\_groups) | Additional EKS managed node groups definition. | <pre>map(object({<br> ami = optional(string)<br> bootstrap_extra_args = optional(string, "")<br> instance_types = list(string)<br> spot = optional(bool, false)<br> min_per_az = number<br> max_per_az = number<br> desired_per_az = number<br> labels = map(string)<br> taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])<br> tags = optional(map(string), {})<br> volume = object({<br> size = string<br> type = string<br> })<br> }))</pre> | `{}` | no |
| <a name="input_bastion_security_group_id"></a> [bastion\_security\_group\_id](#input\_bastion\_security\_group\_id) | Bastion security group id. | `string` | `""` | no |
| <a name="input_create_bastion_sg"></a> [create\_bastion\_sg](#input\_create\_bastion\_sg) | Create bastion access rules toggle. | `bool` | n/a | yes |
| <a name="input_default_node_groups"></a> [default\_node\_groups](#input\_default\_node\_groups) | EKS managed node groups definition. | <pre>object(<br> {<br> compute = object(<br> {<br> ami = optional(string)<br> instance_types = optional(list(string), ["m5.2xlarge"])<br> spot = optional(bool, false)<br> min_per_az = optional(number, 0)<br> max_per_az = optional(number, 10)<br> desired_per_az = optional(number, 1)<br> labels = optional(map(string), {<br> "dominodatalab.com/node-pool" = "default"<br> })<br> taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])<br> tags = optional(map(string), {})<br> volume = optional(object(<br> {<br> size = optional(number, 100)<br> type = optional(string, "gp3")<br> }),<br> {<br> size = 100<br> type = "gp3"<br> }<br> )<br> }),<br> platform = object(<br> {<br> ami = optional(string)<br> instance_types = optional(list(string), ["m5.4xlarge"])<br> spot = optional(bool, false)<br> min_per_az = optional(number, 0)<br> max_per_az = optional(number, 10)<br> desired_per_az = optional(number, 1)<br> labels = optional(map(string), {<br> "dominodatalab.com/node-pool" = "platform"<br> })<br> taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])<br> tags = optional(map(string), {})<br> volume = optional(object(<br> {<br> size = optional(number, 100)<br> type = optional(string, "gp3")<br> }),<br> {<br> size = 100<br> type = "gp3"<br> }<br> )<br> }),<br> gpu = object(<br> {<br> ami = optional(string)<br> instance_types = optional(list(string), ["g4dn.xlarge"])<br> spot = optional(bool, false)<br> min_per_az = optional(number, 0)<br> max_per_az = optional(number, 10)<br> desired_per_az = optional(number, 0)<br> labels = optional(map(string), {<br> "dominodatalab.com/node-pool" = "default-gpu"<br> "nvidia.com/gpu" = true<br> })<br> taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])<br> tags = optional(map(string), {})<br> volume = optional(object(<br> {<br> size = optional(number, 100)<br> type = optional(string, "gp3")<br> }),<br> {<br> size = 100<br> type = "gp3"<br> }<br> )<br> })<br> })</pre> | <pre>{<br> "compute": {},<br> "gpu": {},<br> "platform": {}<br>}</pre> | no |
| <a name="input_default_node_groups"></a> [default\_node\_groups](#input\_default\_node\_groups) | EKS managed node groups definition. | <pre>object(<br> {<br> compute = object(<br> {<br> ami = optional(string)<br> bootstrap_extra_args = optional(string, "")<br> instance_types = optional(list(string), ["m5.2xlarge"])<br> spot = optional(bool, false)<br> min_per_az = optional(number, 0)<br> max_per_az = optional(number, 10)<br> desired_per_az = optional(number, 1)<br> labels = optional(map(string), {<br> "dominodatalab.com/node-pool" = "default"<br> })<br> taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])<br> tags = optional(map(string), {})<br> volume = optional(object(<br> {<br> size = optional(number, 100)<br> type = optional(string, "gp3")<br> }),<br> {<br> size = 100<br> type = "gp3"<br> }<br> )<br> }),<br> platform = object(<br> {<br> ami = optional(string)<br> bootstrap_extra_args = optional(string, "")<br> instance_types = optional(list(string), ["m5.4xlarge"])<br> spot = optional(bool, false)<br> min_per_az = optional(number, 0)<br> max_per_az = optional(number, 10)<br> desired_per_az = optional(number, 1)<br> labels = optional(map(string), {<br> "dominodatalab.com/node-pool" = "platform"<br> })<br> taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])<br> tags = optional(map(string), {})<br> volume = optional(object(<br> {<br> size = optional(number, 100)<br> type = optional(string, "gp3")<br> }),<br> {<br> size = 100<br> type = "gp3"<br> }<br> )<br> }),<br> gpu = object(<br> {<br> ami = optional(string)<br> bootstrap_extra_args = optional(string, "")<br> instance_types = optional(list(string), ["g4dn.xlarge"])<br> spot = optional(bool, false)<br> min_per_az = optional(number, 0)<br> max_per_az = optional(number, 10)<br> desired_per_az = optional(number, 0)<br> labels = optional(map(string), {<br> "dominodatalab.com/node-pool" = "default-gpu"<br> "nvidia.com/gpu" = true<br> })<br> taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])<br> tags = optional(map(string), {})<br> volume = optional(object(<br> {<br> size = optional(number, 100)<br> type = optional(string, "gp3")<br> }),<br> {<br> size = 100<br> type = "gp3"<br> }<br> )<br> })<br> })</pre> | <pre>{<br> "compute": {},<br> "gpu": {},<br> "platform": {}<br>}</pre> | no |
| <a name="input_deploy_id"></a> [deploy\_id](#input\_deploy\_id) | Domino Deployment ID | `string` | n/a | yes |
| <a name="input_efs_security_group"></a> [efs\_security\_group](#input\_efs\_security\_group) | Security Group ID for EFS | `string` | n/a | yes |
| <a name="input_eks_cluster_addons"></a> [eks\_cluster\_addons](#input\_eks\_cluster\_addons) | EKS cluster addons. | `list(string)` | <pre>[<br> "vpc-cni",<br> "kube-proxy",<br> "coredns"<br>]</pre> | no |
Expand Down
18 changes: 16 additions & 2 deletions submodules/eks/node-group.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,22 @@ resource "aws_launch_template" "node_groups" {
name = "${local.eks_cluster_name}-${each.key}"
disable_api_termination = false
key_name = var.ssh_pvt_key_path
vpc_security_group_ids = [aws_security_group.eks_nodes.id]
image_id = each.value.ami
user_data = each.value.ami == null ? null : base64encode(templatefile(
"${path.module}/templates/linux_user_data.tpl",
{
# https://docs.aws.amazon.com/eks/latest/userguide/launch-templates.html#launch-template-custom-ami
# Required to bootstrap node
cluster_name = aws_eks_cluster.this.name
cluster_endpoint = aws_eks_cluster.this.endpoint
cluster_auth_base64 = aws_eks_cluster.this.certificate_authority[0].data
# Optional
cluster_service_ipv4_cidr = aws_eks_cluster.this.kubernetes_network_config[0].service_ipv4_cidr != null ? aws_eks_cluster.this.kubernetes_network_config[0].service_ipv4_cidr : ""
bootstrap_extra_args = each.value.bootstrap_extra_args
pre_bootstrap_user_data = ""
post_bootstrap_user_data = ""
}))
vpc_security_group_ids = [aws_security_group.eks_nodes.id]
image_id = each.value.ami

block_device_mappings {
device_name = "/dev/xvda"
Expand Down
10 changes: 10 additions & 0 deletions submodules/eks/templates/linux_user_data.tpl
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#!/bin/bash
set -e
${pre_bootstrap_user_data ~}
%{ if length(cluster_service_ipv4_cidr) > 0 ~}
export SERVICE_IPV4_CIDR=${cluster_service_ipv4_cidr}
%{ endif ~}
B64_CLUSTER_CA=${cluster_auth_base64}
API_SERVER_URL=${cluster_endpoint}
/etc/eks/bootstrap.sh ${cluster_name} ${bootstrap_extra_args} --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL
${post_bootstrap_user_data ~}
Loading

0 comments on commit e44fc56

Please sign in to comment.