Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

convert some jobs to Ginkgo --label-filter #32648

Merged
merged 2 commits into from
Jun 16, 2024
Merged

convert some jobs to Ginkgo --label-filter #32648

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0edb8ca
SIG node: convert jobs to Ginkgo --label-filter
pohly May 23, 2024
3d87520
SIG testing: convert alpha/beta jobs to --label-filter
pohly May 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions config/jobs/kubernetes/sig-node/dynamic-resource-allocation.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ periodics:
kind build node-image --image=dra/node:latest . &&
trap 'kind export logs "${ARTIFACTS}/kind"; kind delete cluster' EXIT &&
kind create cluster --retain --config test/e2e/dra/kind.yaml --image dra/node:latest &&
KUBERNETES_PROVIDER=local KUBECONFIG=${HOME}/.kube/config GINKGO_PARALLEL_NODES=8 E2E_REPORT_DIR=${ARTIFACTS} hack/ginkgo-e2e.sh -ginkgo.focus=DynamicResourceAllocation -ginkgo.skip=\[Serial\]
KUBERNETES_PROVIDER=local KUBECONFIG=${HOME}/.kube/config GINKGO_PARALLEL_NODES=8 E2E_REPORT_DIR=${ARTIFACTS} hack/ginkgo-e2e.sh -ginkgo.filter='Feature: containsAny DynamicResourceAllocation && !Serial'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Of course, now that I look at this one last time after merging I am spotting a typo! 🥵

s/-ginkgo.filter/-gingko.label-filter/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#32770


# docker-in-docker needs privileged mode
securityContext:
Expand Down Expand Up @@ -82,7 +82,7 @@ periodics:
- '--node-test-args=--feature-gates="DynamicResourceAllocation=true" --service-feature-gates="DynamicResourceAllocation=true" --runtime-config=resource.k8s.io/v1alpha2=true --container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}"'
- --node-tests=true
- --provider=gce
- --test_args=--focus="\[Feature:DynamicResourceAllocation\]" --skip="\[Flaky\]"
- "--test_args=--label-filter='Feature: containsAny DynamicResourceAllocation && !Flaky'"
- --timeout=65m
- --node-args=--image-config-file=/home/prow/go/src/k8s.io/test-infra/jobs/e2e_node/crio/latest/image-config-cgrpv1-serial.yaml
env:
Expand Down Expand Up @@ -135,7 +135,7 @@ periodics:
- '--node-test-args=--feature-gates="DynamicResourceAllocation=true" --service-feature-gates="DynamicResourceAllocation=true" --runtime-config=resource.k8s.io/v1alpha2=true --container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}"'
- --node-tests=true
- --provider=gce
- --test_args=--focus="\[Feature:DynamicResourceAllocation\]" --skip="\[Flaky\]"
- "--test_args=--label-filter='Feature: containsAny DynamicResourceAllocation && !Flaky'"
- --timeout=65m
- --node-args=--image-config-file=/home/prow/go/src/k8s.io/test-infra/jobs/e2e_node/crio/latest/image-config-cgrpv2-serial.yaml
env:
Expand Down Expand Up @@ -188,7 +188,7 @@ periodics:
- '--node-test-args=--feature-gates="DynamicResourceAllocation=true" --service-feature-gates="DynamicResourceAllocation=true,SchedulerQueueingHints=true" --runtime-config=resource.k8s.io/v1alpha2=true --container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}"'
- --node-tests=true
- --provider=gce
- --test_args=--focus="\[Feature:DynamicResourceAllocation\]" --skip="\[Flaky\]"
- "--test_args=--label-filter='Feature: containsAny DynamicResourceAllocation && !Flaky'"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

e2e-node tests do not preserve white space properly 😢

From https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/124548/pull-kubernetes-unit/1802615829762674688:

I0617 08:30:21.206725    9333 ssh.go:146] Running the command ssh, with args: [-o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o CheckHostIP=no -o StrictHostKeyChecking=no -o ServerAliveInterval=30 -o LogLevel=ERROR -i /workspace/.ssh/google_compute_engine [email protected] -- sudo /bin/bash -c 'cd /tmp/node-e2e-20240617T083009 && set -o pipefail; timeout -k 30s 3900.000000s ./ginkgo --label-filter='Feature: containsAny DynamicResourceAllocation && !Flaky' --no-color -v ./e2e_node.test -- --system-spec-name= --system-spec-file= --extra-envs= --runtime-config= --v 4 --node-name=n1-standard-4-fedora-coreos-40-20240519-3-0-gcp-x86-64-54709c74 --report-dir=/tmp/node-e2e-20240617T083009/results --report-prefix=fedora --image-description="fedora-coreos-40-20240519-3-0-gcp-x86-64" --feature-gates="DynamicResourceAllocation=true" --service-feature-gates="DynamicResourceAllocation=true" --runtime-config=resource.k8s.io/v1alpha2=true --container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}" 2>&1 | tee -i /tmp/node-e2e-20240617T083009/results/n1-standard-4-fedora-coreos-40-20240519-3-0-gcp-x86-64-54709c74-ginkgo.log']
E0617 08:30:21.848227    9333 ssh.go:149] failed to run SSH command: out: �[38;5;9m�[1mginkgo run�[0m �[38;5;9mfailed�[0m
  Found no test suites
, err: exit status 1

In this case, it's a limitation of ssh, but that is something that the e2e_node tests should be aware of. The solution for "run complex shell commands via ssh" is to invoke ssh /bin/sh and then feed it the commands on stdin.

Will prepare a fix.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Except that this occurs so deep down that properly composing the input script has the same problems.

I guess it boils down to "use double quotes" for --test_args - let's try with that: #32774

- --timeout=65m
- --node-args=--image-config-file=/home/prow/go/src/k8s.io/test-infra/jobs/e2e_node/crio/latest/image-config-cgrpv1-serial.yaml
env:
Expand Down Expand Up @@ -240,7 +240,7 @@ periodics:
- '--node-test-args=--feature-gates="DynamicResourceAllocation=true" --service-feature-gates="DynamicResourceAllocation=true,SchedulerQueueingHints=true" --runtime-config=resource.k8s.io/v1alpha2=true --container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}"'
- --node-tests=true
- --provider=gce
- --test_args=--focus="\[Feature:DynamicResourceAllocation\]" --skip="\[Flaky\]"
- "--test_args=--label-filter='Feature: containsAny DynamicResourceAllocation && !Flaky'"
- --timeout=65m
- --node-args=--image-config-file=/home/prow/go/src/k8s.io/test-infra/jobs/e2e_node/crio/latest/image-config-cgrpv2-serial.yaml
env:
Expand Down Expand Up @@ -292,7 +292,7 @@ periodics:
- '--node-test-args=--feature-gates="DynamicResourceAllocation=true" --service-feature-gates="DynamicResourceAllocation=true" --runtime-config=resource.k8s.io/v1alpha2=true --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --container-runtime-process-name=/usr/local/bin/containerd --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/containerd.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"containerd.log\", \"journalctl\": [\"-u\", \"containerd\"]}"'
- --node-tests=true
- --provider=gce
- --test_args=--focus="\[Feature:DynamicResourceAllocation\]" --skip="\[Flaky\]"
- "--test_args=--label-filter='Feature: containsAny DynamicResourceAllocation && !Flaky'"
- --timeout=65m
- --node-args=--image-config-file=/home/prow/go/src/k8s.io/test-infra/jobs/e2e_node/dra/image-config-containerd-1.7.yaml
resources:
Expand Down
21 changes: 11 additions & 10 deletions config/jobs/kubernetes/sig-node/sig-node-presubmit.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -655,7 +655,7 @@ presubmits:
- --provider=gce
# Feature:DynamicResourceAllocation has it's own test jobs with the proper conditions set
# DRA tests are covered here https://testgrid.k8s.io/sig-node-dynamic-resource-allocation
- --test_args=--nodes=1 --timeout=4h --focus="\[Serial\].*\[NodeAlphaFeature:.+\]|\[NodeAlphaFeature:.+\].*\[Serial\]" --skip="\[Flaky\]|\[Benchmark\]|\[NodeSpecialFeature:.+\]|\[NodeSpecialFeature\]|\[NodeFeature:Eviction\]\[Feature:CPUManager\]|\[Feature:MemoryManager\]|\[Feature:TopologyManager\]|\[Feature:DynamicResourceAllocation\]"
- "--test_args=--nodes=1 --timeout=4h --label-filter='Serial && !(NodeAlphaFeature: isEmpty) && !Flaky && !Benchmark && NodeSpecialFeature: isEmpty && !(NodeFeature: containsAny Eviction && Feature: containsAny CPUManager) && !(Feature: containsAny { MemoryManager, TopologyManager, DynamicResourceAllocation })'"
- --timeout=240m
env:
- name: GOPATH
Expand Down Expand Up @@ -702,7 +702,7 @@ presubmits:
- '--node-test-args=--feature-gates=SidecarContainers=true --service-feature-gates=SidecarContainers=true --container-runtime-endpoint=unix:///run/containerd/containerd.sock --container-runtime-process-name=/usr/bin/containerd --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/containerd.service" --extra-log="{\"name\": \"containerd.log\", \"journalctl\": [\"-u\", \"containerd*\"]}"'
- --node-tests=true
- --provider=gce
- --test_args=--nodes=1 --timeout=4h --focus="\[Serial\].*\[NodeFeature:SidecarContainers\]|\[NodeFeature:SidecarContainers\].*\[Serial\]" --skip="\[Flaky\]|\[Benchmark\]|\[NodeSpecialFeature:.+\]|\[NodeSpecialFeature\]|\[NodeFeature:Eviction\]"
- "--test_args=--nodes=1 --timeout=4h --label-filter='Serial && NodeFeature: containsAny SidecarContainers && !Flaky && !Benchmark && NodeSpecialFeature: isEmpty && !(NodeFeature: containsAny Eviction)'"
- --timeout=240m
env:
- name: GOPATH
Expand Down Expand Up @@ -812,7 +812,7 @@ presubmits:
- --node-test-args=--kubelet-flags="--cgroups-per-qos=true --cgroup-root=/ --cgroup-driver=systemd"
- --node-tests=true
- --provider=gce
- --test_args=--nodes=1 --skip="" --focus="\[Feature:CPUManager\]"
- "--test_args=--nodes=1 --skip= --label-filter='Feature: containsAny CPUManager'"
- --timeout=180m
env:
- name: GOPATH
Expand Down Expand Up @@ -869,6 +869,7 @@ presubmits:
- --repo-root=.
- --gcp-zone=us-west1-b
- --parallelism=1
# TODO: enable --label-filter in kubetest2
- --focus-regex=\[Feature:CPUManager\]
- --skip-regex=\[Flaky\]|\[Slow\]
- --test-args=--kubelet-flags="--cgroups-per-qos=true --cgroup-root=/ --cgroup-driver=systemd"
Expand Down Expand Up @@ -919,7 +920,7 @@ presubmits:
- '--node-test-args=--container-runtime-endpoint=unix:///run/containerd/containerd.sock --container-runtime-process-name=/usr/bin/containerd --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/containerd.service" --extra-log="{\"name\": \"containerd.log\", \"journalctl\": [\"-u\", \"containerd*\"]}"'
- --node-tests=true
- --provider=gce
- --test_args=--nodes=1 --skip="" --focus="\[Feature:TopologyManager\]"
- "--test_args=--nodes=1 --label-filter='Feature: containsAny TopologyManager'"
- --timeout=180m
env:
- name: GOPATH
Expand Down Expand Up @@ -1025,7 +1026,7 @@ presubmits:
- '--node-test-args=--kubelet-flags="--cgroups-per-qos=true --cgroup-root=/" --extra-log="{\"name\": \"containerd.log\", \"journalctl\": [\"-u\", \"containerd*\"]}"'
- --node-tests=true
- --provider=gce
- --test_args=--nodes=1 --skip="" --focus="\[Feature:HugePages\]"
- "--test_args=--nodes=1 --label-filter='Feature: containsAny HugePages'"
- --timeout=180m
env:
- name: GOPATH
Expand Down Expand Up @@ -1067,7 +1068,7 @@ presubmits:
- '--node-test-args=--container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}"'
- --node-tests=true
- --provider=gce
- --test_args=--nodes=8 --focus="\[NodeConformance\]|\[NodeFeature:.+\]|\[NodeFeature\]" --skip="\[Flaky\]|\[Slow\]|\[Serial\]"
- "--test_args=--nodes=8 --label-filter='(NodeConformance || !(NodeFeature: isEmpty)) && !Flaky && !Slow && !Serial'"
- --timeout=180m
- --node-args=--image-config-file=/home/prow/go/src/k8s.io/test-infra/jobs/e2e_node/crio/latest/image-config-cgrpv2.yaml
resources:
Expand Down Expand Up @@ -2272,7 +2273,7 @@ presubmits:
kind build node-image --image=dra/node:latest . &&
trap 'kind export logs "${ARTIFACTS}/kind"; kind delete cluster' EXIT &&
kind create cluster --retain --config test/e2e/dra/kind.yaml --image dra/node:latest &&
KUBERNETES_PROVIDER=local KUBECONFIG=${HOME}/.kube/config GINKGO_PARALLEL_NODES=8 E2E_REPORT_DIR=${ARTIFACTS} hack/ginkgo-e2e.sh -ginkgo.focus=DynamicResourceAllocation -ginkgo.skip=\[Serial\]
KUBERNETES_PROVIDER=local KUBECONFIG=${HOME}/.kube/config GINKGO_PARALLEL_NODES=8 E2E_REPORT_DIR=${ARTIFACTS} hack/ginkgo-e2e.sh -ginkgo.label-filter='Feature: containsAny DynamicResourceAllocation && !Flaky && !Serial'

# docker-in-docker needs privileged mode
securityContext:
Expand Down Expand Up @@ -3121,7 +3122,7 @@ presubmits:
- '--node-test-args=--feature-gates="DynamicResourceAllocation=true" --service-feature-gates="DynamicResourceAllocation=true" --runtime-config=resource.k8s.io/v1alpha2=true --container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}"'
- --node-tests=true
- --provider=gce
- --test_args=--focus="\[Feature:DynamicResourceAllocation\]" --skip="\[Flaky\]"
- "--test_args=--label-filter='Feature: containsAny DynamicResourceAllocation && !Flaky'"
- --timeout=65m
- --node-args=--image-config-file=/home/prow/go/src/k8s.io/test-infra/jobs/e2e_node/crio/latest/image-config-cgrpv1-serial.yaml
env:
Expand Down Expand Up @@ -3173,7 +3174,7 @@ presubmits:
- '--node-test-args=--feature-gates="DynamicResourceAllocation=true" --service-feature-gates="DynamicResourceAllocation=true" --runtime-config=resource.k8s.io/v1alpha2=true --container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}"'
- --node-tests=true
- --provider=gce
- --test_args=--focus="\[Feature:DynamicResourceAllocation\]" --skip="\[Flaky\]"
- "--test_args=--label-filter='Feature: containsAny DynamicResourceAllocation && !Flaky'"
- --timeout=65m
- --node-args=--image-config-file=/home/prow/go/src/k8s.io/test-infra/jobs/e2e_node/crio/latest/image-config-cgrpv2-serial.yaml
env:
Expand Down Expand Up @@ -3222,7 +3223,7 @@ presubmits:
- '--node-test-args=--feature-gates="DynamicResourceAllocation=true" --service-feature-gates="DynamicResourceAllocation=true" --runtime-config=resource.k8s.io/v1alpha2=true --container-runtime-endpoint=unix:///run/containerd/containerd.sock --container-runtime-process-name=/usr/bin/containerd --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/containerd.service" --extra-log="{\"name\": \"containerd.log\", \"journalctl\": [\"-u\", \"containerd\"]}"'
- --node-tests=true
- --provider=gce
- --test_args=--focus="\[Feature:DynamicResourceAllocation\]" --skip="\[Flaky\]"
- "--test_args=--label-filter='Feature: containsAny DynamicResourceAllocation && !Flaky'"
- --timeout=65m
- --node-args=--image-config-file=/home/prow/go/src/k8s.io/test-infra/jobs/e2e_node/dra/image-config-containerd-1.7.yaml
resources:
Expand Down
Loading