Ensure etcd stateful set has required capabilities to run on OpenShift #2069
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Dom Del Nano <[email protected]>
aimichelle
approved these changes
Dec 23, 2024
2 tasks
aimichelle
pushed a commit
that referenced
this pull request
Jan 1, 2025
…2070) Summary: Ensure securityContext key is inside StatefulSet's containers block While testing the [0.14.14-pre-r1.0 Vizier release](https://github.com/pixie-io/pixie/releases/tag/release%2Fvizier%2Fv0.14.14-pre-r1.0), I realized that the fix from #2069 wasn't applied properly. The `securityContext` block was indented too far and results in an invalid manifest. Relevant Issues: N/A Type of change: /kind bug Test Plan: Verified the following - [x] Existing yamls produce error if applied as is ``` $ kubectl -n pl apply -k k8s/vizier_deps/base/etcd The request is invalid: patch: Invalid value: "map[metadata:map[annotations:map[kubectl.kubernetes.io/last-applied-configuration:{\"apiVersion\":\"apps/v1\",\"kind\":\"StatefulSet\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"pl-monitoring\",\"etcd_cluster\":\"pl-etcd\"},\"name\":\"pl-etcd\",\"namespace\":\"pl\"},\"spec\":{\"podManagementPolicy\":\"Parallel\",\"replicas\":3,\"selector\":{\"matchLabels\":{\"app\":\"pl-monitoring\",\"etcd_cluster\":\"pl-etcd\"}},\"serviceName\":\"pl-etcd\",\"template\":{\"metadata\":{\"labels\":{\"app\":\"pl-monitoring\",\"etcd_cluster\":\"pl-etcd\",\"plane\":\"control\"},\"name\":\"pl-etcd\"},\"spec\":{\"containers\":[{\"env\":[{\"name\":\"INITIAL_CLUSTER_SIZE\",\"value\":\"3\"},{\"name\":\"CLUSTER_NAME\",\"value\":\"pl-etcd\"},{\"name\":\"ETCDCTL_API\",\"value\":\"3\"},{\"name\":\"POD_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.namespace\"}}},{\"name\":\"DATA_DIR\",\"value\":\"/var/run/etcd\"},{\"name\":\"ETCD_AUTO_COMPACTION_RETENTION\",\"value\":\"5\"},{\"name\":\"ETCD_AUTO_COMPACTION_MODE\",\"value\":\"revision\"}],\"image\":\"gcr.io/pixie-oss/pixie-dev-public/etcd:3.5.9@sha256:e18afc6dda592b426834342393c4c4bd076cb46fa7e10fa7818952cae3047ca9\",\"lifecycle\":{\"preStop\":{\"exec\":{\"command\":[\"/etc/etcd/scripts/prestop.sh\"]}}},\"livenessProbe\":{\"exec\":{\"command\":[\"/etc/etcd/scripts/healthcheck.sh\"]},\"failureThreshold\":5,\"initialDelaySeconds\":60,\"periodSeconds\":10,\"securityContext\":{\"capabilities\":{\"add\":[\"NET_RAW\"]},\"seccompProfile\":{\"type\":\"RuntimeDefault\"}},\"successThreshold\":1,\"timeoutSeconds\":5},\"name\":\"etcd\",\"ports\":[{\"containerPort\":2379,\"name\":\"client\"},{\"containerPort\":2380,\"name\":\"server\"}],\"readinessProbe\":{\"exec\":{\"command\":[\"/etc/etcd/scripts/healthcheck.sh\"]},\"failureThreshold\":3,\"initialDelaySeconds\":1,\"periodSeconds\":5,\"successThreshold\":1,\"timeoutSeconds\":5},\"volumeMounts\":[{\"mountPath\":\"/var/run/etcd\",\"name\":\"etcd-data\"},{\"mountPath\":\"/etc/etcdtls/member/peer-tls\",\"name\":\"member-peer-tls\"},{\"mountPath\":\"/etc/etcdtls/member/server-tls\",\"name\":\"member-server-tls\"},{\"mountPath\":\"/etc/etcdtls/client/etcd-tls\",\"name\":\"etcd-client-tls\"}]}],\"securityContext\":{\"seccompProfile\":{\"type\":\"RuntimeDefault\"}},\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"kubernetes.io/arch\",\"operator\":\"Equal\",\"value\":\"amd64\"},{\"effect\":\"NoExecute\",\"key\":\"kubernetes.io/arch\",\"operator\":\"Equal\",\"value\":\"amd64\"},{\"effect\":\"NoSchedule\",\"key\":\"kubernetes.io/arch\",\"operator\":\"Equal\",\"value\":\"arm64\"},{\"effect\":\"NoExecute\",\"key\":\"kubernetes.io/arch\",\"operator\":\"Equal\",\"value\":\"arm64\"}],\"volumes\":[{\"name\":\"member-peer-tls\",\"secret\":{\"secretName\":\"etcd-peer-tls-certs\"}},{\"name\":\"member-server-tls\",\"secret\":{\"secretName\":\"etcd-server-tls-certs\"}},{\"name\":\"etcd-client-tls\",\"secret\":{\"secretName\":\"etcd-client-tls-certs\"}},{\"emptyDir\":{},\"name\":\"etcd-data\"}]}}}}\n]] spec:map[template:map[spec:map[tolerations:[map[effect:NoSchedule key:kubernetes.io/arch operator:Equal value:amd64] map[effect:NoExecute key:kubernetes.io/arch operator:Equal value:amd64] map[effect:NoSchedule key:kubernetes.io/arch operator:Equal value:arm64] map[effect:NoExecute key:kubernetes.io/arch operator:Equal value:arm64]]]]]]": strict decoding error: unknown field "spec.template.spec.containers[0].livenessProbe.securityContext" ``` - [x] Manifests from this PR deploy etcd properly ``` $ git diff diff --git a/k8s/vizier_deps/base/etcd/etcd_statefulset.yaml b/k8s/vizier_deps/base/etcd/etcd_statefulset.yaml index 01d6a6c71..0f4452c1a 100644 --- a/k8s/vizier_deps/base/etcd/etcd_statefulset.yaml +++ b/k8s/vizier_deps/base/etcd/etcd_statefulset.yaml @@ -106,12 +106,12 @@ spec: periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 - securityContext: - capabilities: - add: - - NET_RAW - seccompProfile: - type: RuntimeDefault + securityContext: + capabilities: + add: + - NET_RAW + seccompProfile: + type: RuntimeDefault volumeMounts: - mountPath: /var/run/etcd name: etcd-data $ kubectl -n pl apply -k k8s/vizier_deps/base/etcd service/pl-etcd unchanged service/pl-etcd-client unchanged Warning: resource statefulsets/pl-etcd is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically. statefulset.apps/pl-etcd configured poddisruptionbudget.policy/pl-etcd-pdb configured ``` Signed-off-by: Dom Del Nano <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: Ensure etcd stateful set has required capabilities to run on OpenShift
When using the etcd metadata store on an Openshift cluster, the container gets stuck in its start up script and continuously prints the following error.
The etcd stateful set requires an additional capability, which was missed when the other services had stricter security context settings added. This change also requires the following
SecurityContextConstraintschanges (pixie-io/docs.px.dev#292)Relevant Issues: N/A
Type of change: /kind bug
Test Plan: Deployed the non-operator version of Pixie to an Openshift cluster and verified etcd is scheduled now
Changelog Message: Fixed an issue where the etcd metadata store wouldn't schedule on Openshift clusters