From adfa9dd8c0e1111690e4c3294f3faeff1fd7ddac Mon Sep 17 00:00:00 2001 From: Daniel Gibson Date: Sun, 8 Oct 2023 17:14:23 -0500 Subject: [PATCH] Docs to explain merge_behavior: DEEP in k8s configuration --- .../customizing-your-deployment.mdx | 24 ++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/docs/content/deployment/guides/kubernetes/customizing-your-deployment.mdx b/docs/content/deployment/guides/kubernetes/customizing-your-deployment.mdx index 41113d3f39a03..aac7006f70515 100644 --- a/docs/content/deployment/guides/kubernetes/customizing-your-deployment.mdx +++ b/docs/content/deployment/guides/kubernetes/customizing-your-deployment.mdx @@ -214,24 +214,42 @@ Other executors will ignore the `dagster-k8s/config` tag when it is set on an op ### Precedence rules -If a Kubernetes configuration dictionary (like `container_config`) is specified at both the instance level in the Helm chart and in a specific Dagster job or op, the dictionaries will be shallowly merged. The more specific configuration takes precedence if the same key is set in both dictionaries. +By default, if a Kubernetes configuration dictionary (like `container_config`) is specified at both the instance level in the Helm chart and in a specific Dagster job or op, the dictionaries will be shallowly merged. The more specific configuration takes precedence if the same key is set in both dictionaries. Consider the following example: - **In the Helm chart**, `k8sRunLauncher.runK8sConfig.podSpecConfig` is set to: ```json - { "nodeSelector": { "disktype": "ssd" }, "dns_policy": "ClusterFirst" } + { "node_selector": { "disktype": "ssd" }, "dns_policy": "ClusterFirst" } ``` - **But a specific job** has the `pod_spec_config` key in the `dagster-k8s/config` tag set to: ```json - { "nodeSelector": { "region": "east" } } + { "node_selector": { "region": "east" } } ``` Then the node selector from the job and the DNS policy from the Helm chart will be applied, since only the node selector is overridden in the job. +To customize this behavior, you can also set the `merge_behavior` key in the `dagster-k8s/config` tag to `DEEP` instead of `SHALLOW`. When `merge_behavior` is set to `DEEP`, the config dictionaries are merged recursively. Scalar values will still be replaced by the more specific configuration, but any dictionary fields will be combined, and list fields will be appended to each other. + +To modify the previous example: + +- **In the Helm chart**, `k8sRunLauncher.runK8sConfig.podSpecConfig` is again set to: + + ```json + { "node_selector": { "disktype": "ssd" }, "dns_policy": "ClusterFirst" } + ``` + +- **But a specific job** has the `pod_spec_config` key in the `dagster-k8s/config` tag set to: + + ```json + { "node_selector": { "region": "east" }, "merge_behavior": "DEEP" } + ``` + +Then the job will merge the two `node_selector` dictionaries and use `{ "disktype": "ssd", "region": "east" }` as its `node_selector` configuration. + --- ## Configuring an external database