convert some jobs to Ginkgo --label-filter #32648

pohly · 2024-05-23T16:22:22Z

The new --label-filter expression is easier.

Depends on:

pohly · 2024-05-23T16:23:25Z

config/jobs/kubernetes/sig-node/sig-node-presubmit.yaml

@@ -1067,7 +1068,7 @@ presubmits:
        - '--node-test-args=--container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}"'
        - --node-tests=true
        - --provider=gce
-        - --test_args=--nodes=8 --focus="\[NodeConformance\]|\[NodeFeature:.+\]|\[NodeFeature\]" --skip="\[Flaky\]|\[Slow\]|\[Serial\]"
+        - --test_args=--nodes=8 --label-filter='(NodeConformance || !(NodeFeature: isEmpty)) && !Flaky && !Slow && !Serial'


This ran 353 tests with old and new args.

pohly · 2024-05-23T16:26:35Z

config/jobs/kubernetes/sig-node/sig-node-presubmit.yaml

@@ -702,7 +702,7 @@ presubmits:
          - '--node-test-args=--feature-gates=SidecarContainers=true --service-feature-gates=SidecarContainers=true --container-runtime-endpoint=unix:///run/containerd/containerd.sock --container-runtime-process-name=/usr/bin/containerd --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/containerd.service" --extra-log="{\"name\": \"containerd.log\", \"journalctl\": [\"-u\", \"containerd*\"]}"'
          - --node-tests=true
          - --provider=gce
-          - --test_args=--nodes=1 --timeout=4h --focus="\[Serial\].*\[NodeFeature:SidecarContainers\]|\[NodeFeature:SidecarContainers\].*\[Serial\]" --skip="\[Flaky\]|\[Benchmark\]|\[NodeSpecialFeature:.+\]|\[NodeSpecialFeature\]|\[NodeFeature:Eviction\]"
+          - --test_args=--nodes=1 --timeout=4h --label-filter='Serial && NodeFeature: containsAny SidecarContainers && !Flaky && !Benchmark && NodeSpecialFeature: isEmpty && !(NodeFeature: containsAny Eviction)'


4 tests with old and new flags:

kubernetes$ _output/bin/ginkgo --dry-run --silence-skips -v --label-filter='Serial && NodeFeature: containsAny SidecarContainers && !Flaky && !Benchmark && NodeSpecialFeature: isEmpty && !(NodeFeature: containsAny Eviction)' ./test/e2e_node/ ... [sig-node] Device Plugin [Feature:DevicePluginProbe] [NodeFeature:DevicePluginProbe] [Serial] DevicePlugin [Serial] [Disruptive] Can schedule a pod with a restartable init container [NodeFeature:SidecarContainers] [sig-node, Feature:DevicePluginProbe, NodeFeature:DevicePluginProbe, Serial, Disruptive, NodeFeature:SidecarContainers] /nvme/gopath/src/k8s.io/kubernetes/test/e2e_node/device_plugin_test.go:610 • [0.000 seconds] ------------------------------ [sig-node] CPU Manager [Serial] [Feature:CPUManager] With kubeconfig updated with static CPU Manager policy run the CPU Manager tests should not reuse CPUs of restartable init containers [NodeFeature:SidecarContainers] [sig-node, Serial, Feature:CPUManager, NodeFeature:SidecarContainers] /nvme/gopath/src/k8s.io/kubernetes/test/e2e_node/cpu_manager_test.go:713 • [0.000 seconds] ------------------------------ [sig-node] [NodeFeature:SidecarContainers] [Serial] Containers Lifecycle should restart the containers in right order after the node reboot [sig-node, NodeFeature:SidecarContainers, Serial] /nvme/gopath/src/k8s.io/kubernetes/test/e2e_node/container_lifecycle_test.go:3128 • [0.000 seconds] ------------------------------ [sig-node] POD Resources [Serial] [Feature:PodResources] [NodeFeature:PodResources] with SRIOV devices in the system with CPU manager Static policy should return the expected responses [NodeFeature:SidecarContainers] [sig-node, Serial, Feature:PodResources, NodeFeature:PodResources, NodeFeature:SidecarContainers] /nvme/gopath/src/k8s.io/kubernetes/test/e2e_node/podresources_test.go:901 • [0.000 seconds] ------------------------------

nice - this is much easier to read and grok.

Quoting turned out to be a bit tricky: without quoting the string, YAML treats it as an object because of the ":".

Not a big problem, just something to remember...

pohly · 2024-05-23T17:01:19Z

config/jobs/kubernetes/sig-testing/kubernetes-kind.yaml

        - name: SKIP
-          value: \[Slow\]|\[Disruptive\]|\[Flaky\]|\[Feature:.+\]|PodSecurityPolicy|LoadBalancer|load.balancer|In-tree.Volumes.\[Driver:.nfs\]|PersistentVolumes.NFS|Network.should.set.TCP.CLOSE_WAIT.timeout|Simple.pod.should.support.exec.through.an.HTTP.proxy|subPath.should.support.existing|should.provide.basic.identity
+          value: PodSecurityPolicy|LoadBalancer|load.balancer|In-tree.Volumes.\[Driver:.nfs\]|PersistentVolumes.NFS|Network.should.set.TCP.CLOSE_WAIT.timeout|Simple.pod.should.support.exec.through.an.HTTP.proxy|subPath.should.support.existing|should.provide.basic.identity


One small difference is that tests with just alpha or beta feature gate dependency are allowed to run. However, we currently don't have any such test because any such test also has to add a Feature:<feature gate name> label to be skipped in normal jobs.

This runs 2322 tests with the old and new flags.

pohly · 2024-06-13T07:39:52Z

config/jobs/kubernetes/sig-testing/kubernetes-kind.yaml

-      - name: FOCUS
-        value: "."
+      - name: LABEL_FILTER
+        value: !Slow && !Disruptive && !Flaky && Feature: isSubsetOf Alpha


kubernetes-sigs/kind#3582 got merged. Does that mean that it is usable now?

In other words, can we merge this PR?

/cc @aojea @BenTheElder

The new --label-filter expression is easier.

One small difference is that tests with just alpha or beta feature gate dependency are allowed to run. However, we currently don't have any such test because any such test also has to add a Feature:<feature gate name> label to be skipped in normal jobs.

aojea · 2024-06-16T19:05:13Z

/lgtm
/approve

@pohly do you mind sending an update to the mailing list with the existing changes and a brief of explanation of the benefits?

k8s-ci-robot · 2024-06-16T19:05:23Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aojea, pohly

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~config/OWNERS~~ [aojea]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2024-06-16T19:22:26Z

@pohly: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

key dynamic-resource-allocation.yaml using file config/jobs/kubernetes/sig-node/dynamic-resource-allocation.yaml
key sig-node-presubmit.yaml using file config/jobs/kubernetes/sig-node/sig-node-presubmit.yaml
key kubernetes-kind.yaml using file config/jobs/kubernetes/sig-testing/kubernetes-kind.yaml

In response to this:

The new --label-filter expression is easier.

Depends on:

Label sets allow for more expressive label filtering onsi/ginkgo#1420

dependencies: ginkgo v2.19.0, gomega v1.33.1 kubernetes#125045

e2e-k8s.sh: support --ginkgo.label-filter kubernetes-sigs/kind#3582

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

pohly · 2024-06-17T05:48:03Z

config/jobs/kubernetes/sig-node/dynamic-resource-allocation.yaml

@@ -32,7 +32,7 @@ periodics:
          kind build node-image --image=dra/node:latest . &&
          trap 'kind export logs "${ARTIFACTS}/kind"; kind delete cluster' EXIT &&
          kind create cluster --retain --config test/e2e/dra/kind.yaml --image dra/node:latest &&
-          KUBERNETES_PROVIDER=local KUBECONFIG=${HOME}/.kube/config GINKGO_PARALLEL_NODES=8 E2E_REPORT_DIR=${ARTIFACTS} hack/ginkgo-e2e.sh -ginkgo.focus=DynamicResourceAllocation -ginkgo.skip=\[Serial\]
+          KUBERNETES_PROVIDER=local KUBECONFIG=${HOME}/.kube/config GINKGO_PARALLEL_NODES=8 E2E_REPORT_DIR=${ARTIFACTS} hack/ginkgo-e2e.sh -ginkgo.filter='Feature: containsAny DynamicResourceAllocation && !Serial'


Of course, now that I look at this one last time after merging I am spotting a typo! 🥵

s/-ginkgo.filter/-gingko.label-filter/

pohly · 2024-06-17T12:09:07Z

config/jobs/kubernetes/sig-node/dynamic-resource-allocation.yaml

@@ -188,7 +188,7 @@ periodics:
        - '--node-test-args=--feature-gates="DynamicResourceAllocation=true" --service-feature-gates="DynamicResourceAllocation=true,SchedulerQueueingHints=true" --runtime-config=resource.k8s.io/v1alpha2=true --container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}"'
        - --node-tests=true
        - --provider=gce
-        - --test_args=--focus="\[Feature:DynamicResourceAllocation\]" --skip="\[Flaky\]"
+        - "--test_args=--label-filter='Feature: containsAny DynamicResourceAllocation && !Flaky'"


e2e-node tests do not preserve white space properly 😢

From https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/124548/pull-kubernetes-unit/1802615829762674688:

I0617 08:30:21.206725 9333 ssh.go:146] Running the command ssh, with args: [-o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o CheckHostIP=no -o StrictHostKeyChecking=no -o ServerAliveInterval=30 -o LogLevel=ERROR -i /workspace/.ssh/google_compute_engine [email protected] -- sudo /bin/bash -c 'cd /tmp/node-e2e-20240617T083009 && set -o pipefail; timeout -k 30s 3900.000000s ./ginkgo --label-filter='Feature: containsAny DynamicResourceAllocation && !Flaky' --no-color -v ./e2e_node.test -- --system-spec-name= --system-spec-file= --extra-envs= --runtime-config= --v 4 --node-name=n1-standard-4-fedora-coreos-40-20240519-3-0-gcp-x86-64-54709c74 --report-dir=/tmp/node-e2e-20240617T083009/results --report-prefix=fedora --image-description="fedora-coreos-40-20240519-3-0-gcp-x86-64" --feature-gates="DynamicResourceAllocation=true" --service-feature-gates="DynamicResourceAllocation=true" --runtime-config=resource.k8s.io/v1alpha2=true --container-runtime-endpoint=unix:///var/run/crio/crio.sock --container-runtime-process-name=/usr/local/bin/crio --container-runtime-pid-file= --kubelet-flags="--cgroup-driver=systemd --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/system.slice/crio.service --kubelet-cgroups=/system.slice/kubelet.service" --extra-log="{\"name\": \"crio.log\", \"journalctl\": [\"-u\", \"crio\"]}" 2>&1 | tee -i /tmp/node-e2e-20240617T083009/results/n1-standard-4-fedora-coreos-40-20240519-3-0-gcp-x86-64-54709c74-ginkgo.log'] E0617 08:30:21.848227 9333 ssh.go:149] failed to run SSH command: out: �[38;5;9m�[1mginkgo run�[0m �[38;5;9mfailed�[0m Found no test suites , err: exit status 1

In this case, it's a limitation of ssh, but that is something that the e2e_node tests should be aware of. The solution for "run complex shell commands via ssh" is to invoke ssh /bin/sh and then feed it the commands on stdin.

Will prepare a fix.

Except that this occurs so deep down that properly composing the input script has the same problems.

I guess it boils down to "use double quotes" for --test_args - let's try with that: #32774

k8s-ci-robot requested review from bart0sh and kannon92 May 23, 2024 16:22

pohly commented May 23, 2024

View reviewed changes

pohly mentioned this pull request Jun 11, 2024

e2e-tests.md: update description of labels kubernetes/community#7824

Merged

4 tasks

This was referenced Jun 12, 2024

WIP: remove unneeded e2e features kubernetes/kubernetes#125452

Open

remove the dedicated job which is used to test storage alpha feature #32741

Open

pohly commented Jun 13, 2024

View reviewed changes

k8s-ci-robot requested review from aojea and BenTheElder June 13, 2024 07:39

pohly changed the title ~~WIP: convert jobs to Ginkgo --label-filter~~ convert some jobs to Ginkgo --label-filter Jun 13, 2024

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 13, 2024

pohly added 2 commits June 13, 2024 10:04

SIG node: convert jobs to Ginkgo --label-filter

0edb8ca

The new --label-filter expression is easier.

pohly force-pushed the ginkgo-label-filter branch from b716288 to 3d87520 Compare June 13, 2024 08:04

k8s-ci-robot assigned aojea Jun 16, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 16, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 16, 2024

k8s-ci-robot merged commit 128a653 into kubernetes:master Jun 16, 2024
7 checks passed

pohly commented Jun 17, 2024

View reviewed changes

pohly mentioned this pull request Jun 17, 2024

scheduler_perf: add DRA structured parameters test with shared claims kubernetes/kubernetes#124548

Merged

pohly commented Jun 17, 2024

View reviewed changes

carlory mentioned this pull request Jun 28, 2024

sig-testing: uses LABEL_FILTER instead of FOCUS #32867

Merged

carlory mentioned this pull request Jul 5, 2024

convert some jobs to Ginkgo --label-filter #32911

Open

21 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

convert some jobs to Ginkgo --label-filter #32648

convert some jobs to Ginkgo --label-filter #32648

pohly commented May 23, 2024 •

edited

Loading

pohly May 23, 2024

pohly May 23, 2024

onsi May 23, 2024

aojea Jun 11, 2024

pohly Jun 13, 2024

pohly May 23, 2024 •

edited

Loading

pohly Jun 13, 2024

aojea commented Jun 16, 2024

k8s-ci-robot commented Jun 16, 2024

k8s-ci-robot commented Jun 16, 2024

pohly Jun 17, 2024

pohly Jun 17, 2024

pohly Jun 17, 2024

pohly Jun 17, 2024

convert some jobs to Ginkgo --label-filter #32648

convert some jobs to Ginkgo --label-filter #32648

Conversation

pohly commented May 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pohly May 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aojea commented Jun 16, 2024

k8s-ci-robot commented Jun 16, 2024

k8s-ci-robot commented Jun 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pohly commented May 23, 2024 •

edited

Loading

pohly May 23, 2024 •

edited

Loading