Skip to content

Commit

Permalink
Increasing the default volume size for the high workload node class.
Browse files Browse the repository at this point in the history
  • Loading branch information
@LucaCinquini
LucaCinquini committed Sep 10, 2024
1 parent f54f014 commit 14f9962
Show file tree
Hide file tree
Showing 3 changed files with 4 additions and 4 deletions.
4 changes: 2 additions & 2 deletions airflow/dags/cwl_dag.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@
"request_memory": Param(
"4Gi",
type="string",
enum=["8Gi", "16Gi", "32Gi", "64Gi", "128Gi", "256Gi"],
enum=["4Gi", "8Gi", "16Gi", "32Gi", "64Gi", "128Gi", "256Gi"],
title="Docker container memory",
),
"request_cpu": Param(
Expand All @@ -111,7 +111,7 @@
"request_storage": Param(
"10Gi",
type="string",
enum=["10Gi", "50Gi", "100Gi", "200Gi", "300Gi"],
enum=["10Gi", "50Gi", "100Gi", "150Gi", "200Gi", "250Gi"],
title="Docker container storage",
),
"use_ecr": Param(False, type="boolean", title="Log into AWS Elastic Container Registry (ECR)"),
Expand Down
2 changes: 1 addition & 1 deletion terraform-unity/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ terraform apply -no-color 2>&1 | tee apply_output.txt
| <a name="input_airflow_webserver_username"></a> [airflow\_webserver\_username](#input\_airflow\_webserver\_username) | The username for the Airflow webserver and UI. | `string` | `"admin"` | no |
| <a name="input_dag_catalog_repo"></a> [dag\_catalog\_repo](#input\_dag\_catalog\_repo) | Git repository that stores the catalog of Airflow DAGs. | <pre>object({<br> url = string<br> ref = string<br> dags_directory_path = string<br> })</pre> | <pre>{<br> "dags_directory_path": "airflow/dags",<br> "ref": "develop",<br> "url": "https://github.com/unity-sds/unity-sps.git"<br>}</pre> | no |
| <a name="input_helm_charts"></a> [helm\_charts](#input\_helm\_charts) | Helm charts for the associated services. | <pre>map(object({<br> repository = string<br> chart = string<br> version = string<br> }))</pre> | <pre>{<br> "airflow": {<br> "chart": "airflow",<br> "repository": "https://airflow.apache.org",<br> "version": "1.13.1"<br> },<br> "keda": {<br> "chart": "keda",<br> "repository": "https://kedacore.github.io/charts",<br> "version": "v2.14.2"<br> }<br>}</pre> | no |
| <a name="input_karpenter_node_classes"></a> [karpenter\_node\_classes](#input\_karpenter\_node\_classes) | Configuration for karpenter\_node\_classes | <pre>map(object({<br> volume_size = string<br> }))</pre> | <pre>{<br> "airflow-kubernetes-pod-operator-high-workload": {<br> "volume_size": "200Gi"<br> },<br> "default": {<br> "volume_size": "30Gi"<br> }<br>}</pre> | no |
| <a name="input_karpenter_node_classes"></a> [karpenter\_node\_classes](#input\_karpenter\_node\_classes) | Configuration for karpenter\_node\_classes | <pre>map(object({<br> volume_size = string<br> }))</pre> | <pre>{<br> "airflow-kubernetes-pod-operator-high-workload": {<br> "volume_size": "300Gi"<br> },<br> "default": {<br> "volume_size": "30Gi"<br> }<br>}</pre> | no |
| <a name="input_karpenter_node_pools"></a> [karpenter\_node\_pools](#input\_karpenter\_node\_pools) | Configuration for Karpenter node pools | <pre>map(object({<br> requirements : list(object({<br> key : string<br> operator : string<br> values : list(string)<br> }))<br> nodeClassRef : string<br> limits : object({<br> cpu : string<br> memory : string<br> })<br> disruption : object({<br> consolidationPolicy : string<br> consolidateAfter : string<br> })<br> }))</pre> | <pre>{<br> "airflow-celery-workers": {<br> "disruption": {<br> "consolidateAfter": "1m",<br> "consolidationPolicy": "WhenEmpty"<br> },<br> "limits": {<br> "cpu": "80",<br> "memory": "320Gi"<br> },<br> "nodeClassRef": "default",<br> "requirements": [<br> {<br> "key": "karpenter.k8s.aws/instance-family",<br> "operator": "In",<br> "values": [<br> "m7i",<br> "m6i",<br> "m5",<br> "t3",<br> "c7i",<br> "c6i",<br> "c5",<br> "r7i",<br> "r6i",<br> "r5"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-cpu",<br> "operator": "Gt",<br> "values": [<br> "1"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-cpu",<br> "operator": "Lt",<br> "values": [<br> "9"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-memory",<br> "operator": "Gt",<br> "values": [<br> "8191"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-memory",<br> "operator": "Lt",<br> "values": [<br> "32769"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-hypervisor",<br> "operator": "In",<br> "values": [<br> "nitro"<br> ]<br> }<br> ]<br> },<br> "airflow-core-components": {<br> "disruption": {<br> "consolidateAfter": "1m",<br> "consolidationPolicy": "WhenEmpty"<br> },<br> "limits": {<br> "cpu": "40",<br> "memory": "160Gi"<br> },<br> "nodeClassRef": "default",<br> "requirements": [<br> {<br> "key": "karpenter.k8s.aws/instance-family",<br> "operator": "In",<br> "values": [<br> "m7i",<br> "m6i",<br> "m5",<br> "t3",<br> "c7i",<br> "c6i",<br> "c5",<br> "r7i",<br> "r6i",<br> "r5"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-cpu",<br> "operator": "Gt",<br> "values": [<br> "1"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-cpu",<br> "operator": "Lt",<br> "values": [<br> "17"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-memory",<br> "operator": "Gt",<br> "values": [<br> "8191"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-memory",<br> "operator": "Lt",<br> "values": [<br> "32769"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-hypervisor",<br> "operator": "In",<br> "values": [<br> "nitro"<br> ]<br> }<br> ]<br> },<br> "airflow-kubernetes-pod-operator": {<br> "disruption": {<br> "consolidateAfter": "1m",<br> "consolidationPolicy": "WhenEmpty"<br> },<br> "limits": {<br> "cpu": "100",<br> "memory": "400Gi"<br> },<br> "nodeClassRef": "default",<br> "requirements": [<br> {<br> "key": "karpenter.k8s.aws/instance-family",<br> "operator": "In",<br> "values": [<br> "m7i",<br> "m6i",<br> "m5",<br> "t3",<br> "c7i",<br> "c6i",<br> "c5",<br> "r7i",<br> "r6i",<br> "r5"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-cpu",<br> "operator": "Gt",<br> "values": [<br> "1"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-cpu",<br> "operator": "Lt",<br> "values": [<br> "17"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-memory",<br> "operator": "Gt",<br> "values": [<br> "8191"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-memory",<br> "operator": "Lt",<br> "values": [<br> "32769"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-hypervisor",<br> "operator": "In",<br> "values": [<br> "nitro"<br> ]<br> }<br> ]<br> },<br> "airflow-kubernetes-pod-operator-high-workload": {<br> "disruption": {<br> "consolidateAfter": "1m",<br> "consolidationPolicy": "WhenEmpty"<br> },<br> "limits": {<br> "cpu": "528",<br> "memory": "1056Gi"<br> },<br> "nodeClassRef": "airflow-kubernetes-pod-operator-high-workload",<br> "requirements": [<br> {<br> "key": "karpenter.k8s.aws/instance-family",<br> "operator": "In",<br> "values": [<br> "m7i",<br> "m6i",<br> "m5",<br> "t3",<br> "c7i",<br> "c6i",<br> "c5",<br> "r7i",<br> "r6i",<br> "r5"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-cpu",<br> "operator": "Gt",<br> "values": [<br> "1"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-cpu",<br> "operator": "Lt",<br> "values": [<br> "49"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-memory",<br> "operator": "Gt",<br> "values": [<br> "8191"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-memory",<br> "operator": "Lt",<br> "values": [<br> "98305"<br> ]<br> },<br> {<br> "key": "karpenter.k8s.aws/instance-hypervisor",<br> "operator": "In",<br> "values": [<br> "nitro"<br> ]<br> }<br> ]<br> }<br>}</pre> | no |
| <a name="input_kubeconfig_filepath"></a> [kubeconfig\_filepath](#input\_kubeconfig\_filepath) | The path to the kubeconfig file for the Kubernetes cluster. | `string` | n/a | yes |
| <a name="input_mcp_ami_owner_id"></a> [mcp\_ami\_owner\_id](#input\_mcp\_ami\_owner\_id) | The owner ID of the MCP AMIs | `string` | `"794625662971"` | no |
Expand Down
2 changes: 1 addition & 1 deletion terraform-unity/variables.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ variable "karpenter_node_classes" {
volume_size = "30Gi"
}
"airflow-kubernetes-pod-operator-high-workload" = {
volume_size = "200Gi"
volume_size = "300Gi"
}
}
}
Expand Down

0 comments on commit 14f9962

Please sign in to comment.