[DOM-54483] Link flyte roles to flyte buckets, add permissions (#218)

* link flyte roles to buckets

* new submodule for flyte

* try hacking infro deploy

* update outputs

* role perm changes

* move flyte module outside of infra

* update docs after merge

* remove kms

* default to AES256 encryption

* add namespace vars to flyte (#221)

* make variable for service account names

* remove one level of nesting for flyte outputs

modules/flyte/README.md

@@ -0,0 +1,59 @@
+# flyte
+
+<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+## Requirements
+
+| Name | Version |
+|------|---------|
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | ~> 5.0 |
+| <a name="requirement_null"></a> [null](#requirement\_null) | >= 3.1.0 |
+
+## Providers
+
+| Name | Version |
+|------|---------|
+| <a name="provider_aws"></a> [aws](#provider\_aws) | ~> 5.0 |
+
+## Modules
+
+No modules.
+
+## Resources
+
+| Name | Type |
+|------|------|
+| [aws_iam_policy.flyte_controlplane](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
+| [aws_iam_policy.flyte_dataplane](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
+| [aws_iam_role.flyte_controlplane](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource |
+| [aws_iam_role.flyte_dataplane](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource |
+| [aws_iam_role_policy_attachment.flyte_controlplane](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
+| [aws_iam_role_policy_attachment.flyte_dataplane](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
+| [aws_s3_bucket.flyte_data](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) | resource |
+| [aws_s3_bucket.flyte_metadata](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) | resource |
+| [aws_s3_bucket_policy.flyte_data](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_policy) | resource |
+| [aws_s3_bucket_policy.flyte_metadata](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_policy) | resource |
+| [aws_s3_bucket_server_side_encryption_configuration.flye_metadata_encryption](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_server_side_encryption_configuration) | resource |
+| [aws_s3_bucket_server_side_encryption_configuration.flyte_data_encryption](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_server_side_encryption_configuration) | resource |
+| [aws_iam_policy_document.flyte_controlplane](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
+| [aws_iam_policy_document.flyte_data](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
+| [aws_iam_policy_document.flyte_dataplane](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
+| [aws_iam_policy_document.flyte_metadata](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
+| [aws_partition.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source |
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| <a name="input_compute_namespace"></a> [compute\_namespace](#input\_compute\_namespace) | Name of Domino compute namespace for this deploy | `string` | n/a | yes |
+| <a name="input_eks_info"></a> [eks\_info](#input\_eks\_info) | cluster = {<br>      specs {<br>        name            = Cluster name.<br>        account\_id      = AWS account id where the cluster resides.<br>      }<br>      oidc = {<br>        arn = OIDC provider ARN.<br>        url = OIDC provider url.<br>        cert = {<br>          thumbprint\_list = OIDC cert thumbprints.<br>          url             = OIDC cert URL.<br>      }<br>    } | <pre>object({<br>    cluster = object({<br>      specs = object({<br>        name       = string<br>        account_id = string<br>      })<br>      oidc = object({<br>        arn = string<br>        url = string<br>        cert = object({<br>          thumbprint_list = list(string)<br>          url             = string<br>        })<br>      })<br>    })<br>  })</pre> | n/a | yes |
+| <a name="input_force_destroy_on_deletion"></a> [force\_destroy\_on\_deletion](#input\_force\_destroy\_on\_deletion) | Whether to force destroy flyte s3 buckets on deletion | `bool` | `true` | no |
+| <a name="input_platform_namespace"></a> [platform\_namespace](#input\_platform\_namespace) | Name of Domino platform namespace for this deploy | `string` | n/a | yes |
+| <a name="input_serviceaccount_names"></a> [serviceaccount\_names](#input\_serviceaccount\_names) | Service account names for Flyte | <pre>object({<br>    datacatalog    = optional(string, "datacatalog")<br>    flyteadmin     = optional(string, "flyteadmin")<br>    flytepropeller = optional(string, "flytepropeller")<br>  })</pre> | `{}` | no |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| <a name="output_eks"></a> [eks](#output\_eks) | Flyte eks info |
+<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

modules/flyte/iam.tf

@@ -0,0 +1,102 @@
+resource "aws_iam_role" "flyte_controlplane" {
+  name = "${local.deploy_id}-flyte-controlplane"
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Action = "sts:AssumeRoleWithWebIdentity"
+        Effect = "Allow"
+        Principal = {
+          Federated = local.oidc_provider_arn
+        }
+        Condition : {
+          StringEquals : {
+            "${trimprefix(local.oidc_provider_url, "https://")}:aud" : "sts.amazonaws.com",
+            "${trimprefix(local.oidc_provider_url, "https://")}:sub" : [
+              "system:serviceaccount:${var.platform_namespace}:${var.serviceaccount_names.flyteadmin}",
+              "system:serviceaccount:${var.platform_namespace}:${var.serviceaccount_names.flytepropeller}",
+            ]
+          }
+        }
+      },
+    ]
+  })
+}
+
+data "aws_iam_policy_document" "flyte_controlplane" {
+  statement {
+    effect = "Allow"
+    resources = [
+      "arn:${data.aws_partition.current.partition}:s3:::${aws_s3_bucket.flyte_metadata.bucket}/*",
+      "arn:${data.aws_partition.current.partition}:s3:::${aws_s3_bucket.flyte_metadata.bucket}"
+    ]
+    actions = [
+      "s3:PutObject",
+      "s3:GetObject",
+      "s3:ListBucket",
+    ]
+  }
+}
+
+resource "aws_iam_policy" "flyte_controlplane" {
+  name   = "${local.deploy_id}-flyte-controlplane"
+  policy = data.aws_iam_policy_document.flyte_controlplane.json
+}
+
+resource "aws_iam_role_policy_attachment" "flyte_controlplane" {
+  role       = aws_iam_role.flyte_controlplane.name
+  policy_arn = aws_iam_policy.flyte_controlplane.arn
+}
+
+resource "aws_iam_role" "flyte_dataplane" {
+  name = "${local.deploy_id}-flyte-dataplane"
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Action = "sts:AssumeRoleWithWebIdentity"
+        Effect = "Allow"
+        Principal = {
+          Federated = local.oidc_provider_arn
+        }
+        Condition : {
+          StringEquals : {
+            "${trimprefix(local.oidc_provider_url, "https://")}:aud" : "sts.amazonaws.com",
+            "${trimprefix(local.oidc_provider_url, "https://")}:sub" : [
+              "system:serviceaccount:${var.platform_namespace}:${var.serviceaccount_names.datacatalog}",
+              "system:serviceaccount:${var.compute_namespace}:*"
+            ]
+          }
+        }
+      },
+    ]
+  })
+}
+
+data "aws_iam_policy_document" "flyte_dataplane" {
+  statement {
+    effect = "Allow"
+    resources = [
+      "arn:${data.aws_partition.current.partition}:s3:::${aws_s3_bucket.flyte_metadata.bucket}/*",
+      "arn:${data.aws_partition.current.partition}:s3:::${aws_s3_bucket.flyte_metadata.bucket}",
+      "arn:${data.aws_partition.current.partition}:s3:::${aws_s3_bucket.flyte_data.bucket}/*",
+      "arn:${data.aws_partition.current.partition}:s3:::${aws_s3_bucket.flyte_data.bucket}",
+    ]
+
+    actions = [
+      "s3:PutObject",
+      "s3:GetObject",
+      "s3:ListBucket"
+    ]
+  }
+}
+
+resource "aws_iam_policy" "flyte_dataplane" {
+  name   = "${local.deploy_id}-flyte-dataplane"
+  policy = data.aws_iam_policy_document.flyte_dataplane.json
+}
+
+resource "aws_iam_role_policy_attachment" "flyte_dataplane" {
+  role       = aws_iam_role.flyte_dataplane.name
+  policy_arn = aws_iam_policy.flyte_dataplane.arn
+}

modules/flyte/main.tf

@@ -0,0 +1,7 @@
+data "aws_partition" "current" {}
+
+locals {
+  deploy_id         = var.eks_info.cluster.specs.name
+  oidc_provider_arn = var.eks_info.cluster.oidc.arn
+  oidc_provider_url = var.eks_info.cluster.oidc.cert.url
+}

modules/flyte/outputs.tf

@@ -0,0 +1,9 @@
+output "eks" {
+  description = "Flyte eks info"
+  value = {
+    metadata_bucket       = aws_s3_bucket.flyte_metadata.bucket
+    data_bucket           = aws_s3_bucket.flyte_data.bucket
+    controlplane_role_arn = aws_iam_role.flyte_controlplane.arn
+    dataplane_role_arn    = aws_iam_role.flyte_dataplane.arn
+  }
+}

modules/flyte/s3.tf

@@ -0,0 +1,101 @@
+resource "aws_s3_bucket" "flyte_metadata" {
+  bucket              = "${local.deploy_id}-flyte-metadata"
+  force_destroy       = var.force_destroy_on_deletion
+  object_lock_enabled = false
+}
+
+data "aws_iam_policy_document" "flyte_metadata" {
+  statement {
+    effect = "Deny"
+
+    resources = [
+      "arn:${data.aws_partition.current.partition}:s3:::${aws_s3_bucket.flyte_metadata.bucket}",
+      "arn:${data.aws_partition.current.partition}:s3:::${aws_s3_bucket.flyte_metadata.bucket}/*",
+    ]
+
+    actions = ["s3:*"]
+
+    condition {
+      test     = "Bool"
+      variable = "aws:SecureTransport"
+      values   = ["false"]
+    }
+
+    principals {
+      type        = "AWS"
+      identifiers = ["*"]
+    }
+  }
+}
+
+resource "aws_s3_bucket_policy" "flyte_metadata" {
+  bucket = aws_s3_bucket.flyte_metadata.id
+  policy = data.aws_iam_policy_document.flyte_metadata.json
+}
+
+resource "aws_s3_bucket_server_side_encryption_configuration" "flye_metadata_encryption" {
+  bucket = aws_s3_bucket.flyte_metadata.bucket
+  rule {
+    apply_server_side_encryption_by_default {
+      sse_algorithm = "AES256"
+    }
+    bucket_key_enabled = false
+  }
+
+  lifecycle {
+    ignore_changes = [
+      rule,
+    ]
+  }
+}
+
+resource "aws_s3_bucket" "flyte_data" {
+  bucket              = "${local.deploy_id}-flyte-data"
+  force_destroy       = var.force_destroy_on_deletion
+  object_lock_enabled = false
+}
+
+data "aws_iam_policy_document" "flyte_data" {
+  statement {
+    effect = "Deny"
+
+    resources = [
+      "arn:${data.aws_partition.current.partition}:s3:::${aws_s3_bucket.flyte_data.bucket}",
+      "arn:${data.aws_partition.current.partition}:s3:::${aws_s3_bucket.flyte_data.bucket}/*",
+    ]
+
+    actions = ["s3:*"]
+
+    condition {
+      test     = "Bool"
+      variable = "aws:SecureTransport"
+      values   = ["false"]
+    }
+
+    principals {
+      type        = "AWS"
+      identifiers = ["*"]
+    }
+  }
+}
+
+resource "aws_s3_bucket_policy" "flyte_data" {
+  bucket = aws_s3_bucket.flyte_data.id
+  policy = data.aws_iam_policy_document.flyte_data.json
+}
+
+resource "aws_s3_bucket_server_side_encryption_configuration" "flyte_data_encryption" {
+  bucket = aws_s3_bucket.flyte_data.bucket
+  rule {
+    apply_server_side_encryption_by_default {
+      sse_algorithm = "AES256"
+    }
+    bucket_key_enabled = false
+  }
+
+  lifecycle {
+    ignore_changes = [
+      rule,
+    ]
+  }
+}

modules/flyte/variables.tf

@@ -0,0 +1,60 @@
+variable "force_destroy_on_deletion" {
+  description = "Whether to force destroy flyte s3 buckets on deletion"
+  type        = bool
+  default     = true
+}
+
+variable "eks_info" {
+  description = <<EOF
+    cluster = {
+      specs {
+        name            = Cluster name.
+        account_id      = AWS account id where the cluster resides.
+      }
+      oidc = {
+        arn = OIDC provider ARN.
+        url = OIDC provider url.
+        cert = {
+          thumbprint_list = OIDC cert thumbprints.
+          url             = OIDC cert URL.
+      }
+    }
+  EOF
+  type = object({
+    cluster = object({
+      specs = object({
+        name       = string
+        account_id = string
+      })
+      oidc = object({
+        arn = string
+        url = string
+        cert = object({
+          thumbprint_list = list(string)
+          url             = string
+        })
+      })
+    })
+  })
+}
+
+variable "platform_namespace" {
+  description = "Name of Domino platform namespace for this deploy"
+  type        = string
+}
+
+variable "compute_namespace" {
+  description = "Name of Domino compute namespace for this deploy"
+  type        = string
+}
+
+variable "serviceaccount_names" {
+  description = "Service account names for Flyte"
+  type = object({
+    datacatalog    = optional(string, "datacatalog")
+    flyteadmin     = optional(string, "flyteadmin")
+    flytepropeller = optional(string, "flytepropeller")
+  })
+
+  default = {}
+}

modules/flyte/versions.tf

@@ -0,0 +1,13 @@
+terraform {
+  required_version = ">= 1.0"
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = ">= 3.1.0"
+    }
+    aws = {
+      source  = "hashicorp/aws"
+      version = "~> 5.0"
+    }
+  }
+}

modules/infra/README.md

@@ -58,7 +58,6 @@
 | <a name="input_deploy_id"></a> [deploy\_id](#input\_deploy\_id) | Domino Deployment ID. | `string` | `"domino-eks"` | no |
 | <a name="input_domino_cur"></a> [domino\_cur](#input\_domino\_cur) | Determines whether to provision domino cost related infrastructures, ie, long term storage | <pre>object({<br>    provision_cost_usage_report = optional(bool, false)<br>  })</pre> | `{}` | no |
 | <a name="input_eks"></a> [eks](#input\_eks) | creation\_role\_name = Name of the role to import.<br>    k8s\_version = EKS cluster k8s version.<br>    nodes\_master  Grants the nodes role system:master access. NOT recomended<br>    kubeconfig = {<br>      extra\_args = Optional extra args when generating kubeconfig.<br>      path       = Fully qualified path name to write the kubeconfig file.<br>    }<br>    public\_access = {<br>      enabled = Enable EKS API public endpoint.<br>      cidrs   = List of CIDR ranges permitted for accessing the EKS public endpoint.<br>    }<br>    Custom role maps for aws auth configmap<br>    custom\_role\_maps = {<br>      rolearn  = string<br>      username = string<br>      groups   = list(string)<br>    }<br>    master\_role\_names  = IAM role names to be added as masters in eks.<br>    cluster\_addons     = EKS cluster addons. vpc-cni is installed separately.<br>    vpc\_cni            = Configuration for AWS VPC CNI<br>    ssm\_log\_group\_name = CloudWatch log group to send the SSM session logs to.<br>    identity\_providers = Configuration for IDP(Identity Provider).<br>  } | <pre>object({<br>    creation_role_name = optional(string, null)<br>    k8s_version        = optional(string, "1.27")<br>    nodes_master       = optional(bool, false)<br>    kubeconfig = optional(object({<br>      extra_args = optional(string, "")<br>      path       = optional(string, null)<br>    }), {})<br>    public_access = optional(object({<br>      enabled = optional(bool, false)<br>      cidrs   = optional(list(string), [])<br>    }), {})<br>    custom_role_maps = optional(list(object({<br>      rolearn  = string<br>      username = string<br>      groups   = list(string)<br>    })), [])<br>    master_role_names  = optional(list(string), [])<br>    cluster_addons     = optional(list(string), ["kube-proxy", "coredns", "vpc-cni"])<br>    ssm_log_group_name = optional(string, "session-manager")<br>    vpc_cni = optional(object({<br>      prefix_delegation = optional(bool)<br>      annotate_pod_ip   = optional(bool)<br>    }))<br>    identity_providers = optional(list(object({<br>      client_id                     = string<br>      groups_claim                  = optional(string, null)<br>      groups_prefix                 = optional(string, null)<br>      identity_provider_config_name = string<br>      issuer_url                    = optional(string, null)<br>      required_claims               = optional(string, null)<br>      username_claim                = optional(string, null)<br>      username_prefix               = optional(string, null)<br>    })), [])<br>  })</pre> | `{}` | no |
-| <a name="input_flyte"></a> [flyte](#input\_flyte) | enabled = Whether to provision any Flyte related resources | <pre>object({<br>    enabled = optional(bool, false)<br>  })</pre> | `{}` | no |
 | <a name="input_ignore_tags"></a> [ignore\_tags](#input\_ignore\_tags) | Tag keys to be ignored by the aws provider. | `list(string)` | `[]` | no |
 | <a name="input_kms"></a> [kms](#input\_kms) | enabled             = "Toggle, if set use either the specified KMS key\_id or a Domino-generated one"<br>    key\_id              = optional(string, null)<br>    additional\_policies = "Allows setting additional KMS key policies when using a Domino-generated key" | <pre>object({<br>    enabled             = optional(bool, true)<br>    key_id              = optional(string, null)<br>    additional_policies = optional(list(string), [])<br>  })</pre> | `{}` | no |
 | <a name="input_network"></a> [network](#input\_network) | vpc = {<br>      id = Existing vpc id, it will bypass creation by this module.<br>      subnets = {<br>        private = Existing private subnets.<br>        public  = Existing public subnets.<br>        pod     = Existing pod subnets.<br>      }), {})<br>    }), {})<br>    network\_bits = {<br>      public  = Number of network bits to allocate to the public subnet. i.e /27 -> 32 IPs.<br>      private = Number of network bits to allocate to the private subnet. i.e /19 -> 8,192 IPs.<br>      pod     = Number of network bits to allocate to the private subnet. i.e /19 -> 8,192 IPs.<br>    }<br>    cidrs = {<br>      vpc     = The IPv4 CIDR block for the VPC.<br>      pod     = The IPv4 CIDR block for the Pod subnets.<br>    }<br>    use\_pod\_cidr = Use additional pod CIDR range (ie 100.64.0.0/16) for pod networking. | <pre>object({<br>    vpc = optional(object({<br>      id = optional(string, null)<br>      subnets = optional(object({<br>        private = optional(list(string), [])<br>        public  = optional(list(string), [])<br>        pod     = optional(list(string), [])<br>      }), {})<br>    }), {})<br>    network_bits = optional(object({<br>      public  = optional(number, 27)<br>      private = optional(number, 19)<br>      pod     = optional(number, 19)<br>      }<br>    ), {})<br>    cidrs = optional(object({<br>      vpc = optional(string, "10.0.0.0/16")<br>      pod = optional(string, "100.64.0.0/16")<br>    }), {})<br>    use_pod_cidr = optional(bool, true)<br>  })</pre> | `{}` | no |

modules/infra/main.tf

@@ -31,7 +31,6 @@ module "storage" {
   network_info = module.network.info
   kms_info     = local.kms_info
   storage      = var.storage
-  flyte        = var.flyte
 }
 
 data "aws_ec2_instance_type" "all" {