🌱 test: fix collector for machines not having an IP in status and change ignition ssh user to capv #3010

chrischdi · 2024-05-16T15:32:31Z

What this PR does / why we need it:

In case of test failures where machines do not yet have an IP set in status, we are currently not able to collect information like cloud-init log.

This implements a fallback to get the IP address from vSphere instead to still be able to gather information via ssh.

Example Job: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-cluster-api-provider-vsphere-e2e-supervisor-conformance-ci-latest-main/1790940294057824256

Which logs:

Failed to get logs for Machine k8s-conformance-9dmcar-cp-z2zcx-mhzxn, Cluster k8s-conformance-n8g9q1/k8s-conformance-9dmcar: dialing host IP address at : dial tcp :22: connect: connection refused

And the resulting empty files: https://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/periodic-cluster-api-provider-vsphere-e2e-supervisor-conformance-ci-latest-main/1790940294057824256/artifacts/clusters/k8s-conformance-9dmcar/machines/k8s-conformance-9dmcar-cp-z2zcx-mhzxn/

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

chrischdi · 2024-05-16T16:43:16Z

/test help

k8s-ci-robot · 2024-05-16T16:43:19Z

@chrischdi: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test pull-cluster-api-provider-vsphere-e2e-govmomi-blocking-main
/test pull-cluster-api-provider-vsphere-e2e-govmomi-conformance-ci-latest-main
/test pull-cluster-api-provider-vsphere-e2e-govmomi-conformance-main
/test pull-cluster-api-provider-vsphere-e2e-govmomi-main
/test pull-cluster-api-provider-vsphere-e2e-govmomi-upgrade-1-30-1-31-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-blocking-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-conformance-ci-latest-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-conformance-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-upgrade-1-30-1-31-main
/test pull-cluster-api-provider-vsphere-e2e-vcsim-govmomi-main
/test pull-cluster-api-provider-vsphere-e2e-vcsim-supervisor-main
/test pull-cluster-api-provider-vsphere-test-main
/test pull-cluster-api-provider-vsphere-verify-main

The following commands are available to trigger optional jobs:

/test pull-cluster-api-provider-vsphere-apidiff-main

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-provider-vsphere-apidiff-main
pull-cluster-api-provider-vsphere-e2e-govmomi-blocking-main
pull-cluster-api-provider-vsphere-e2e-supervisor-blocking-main
pull-cluster-api-provider-vsphere-test-main
pull-cluster-api-provider-vsphere-verify-main

In response to this:

/test help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

chrischdi · 2024-05-16T16:43:28Z

/test pull-cluster-api-provider-vsphere-e2e-govmomi-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-main

chrischdi · 2024-05-17T07:40:40Z

/cherry-pick release-1.10

k8s-infra-cherrypick-robot · 2024-05-17T07:40:42Z

@chrischdi: once the present PR merges, I will cherry-pick it on top of release-1.10 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

chrischdi · 2024-05-17T07:40:42Z

/cherry-pick release-1.9

k8s-infra-cherrypick-robot · 2024-05-17T07:40:44Z

@chrischdi: once the present PR merges, I will cherry-pick it on top of release-1.9 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

chrischdi · 2024-05-17T07:40:45Z

/cherry-pick release-1.8

k8s-infra-cherrypick-robot · 2024-05-17T07:40:47Z

@chrischdi: once the present PR merges, I will cherry-pick it on top of release-1.8 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.8

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

chrischdi · 2024-05-17T07:40:48Z

/cherry-pick release-1.7

k8s-infra-cherrypick-robot · 2024-05-17T07:40:50Z

@chrischdi: once the present PR merges, I will cherry-pick it on top of release-1.7 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

chrischdi · 2024-05-17T07:40:55Z

/assign @sbueringer @fabriziopandini

sbueringer

Nice! just a few nits

test/framework/log/collector.go

sbueringer · 2024-05-17T12:31:32Z

/lgtm

k8s-ci-robot · 2024-05-17T12:31:38Z

LGTM label has been added.

Git tree hash: aa72c221c11c0c44678a604323ebdb3751e0a270

fabriziopandini · 2024-05-20T14:48:27Z

test/e2e/data/infrastructure-vsphere-govmomi/main/ignition/kustomization.yaml

@@ -14,3 +14,10 @@ patches:
  - target:
      kind: VSphereMachineTemplate
    path: ../commons/remove-storage-policy.yaml
+  # Replace ssh user to match expected user by the e2e machine collector


q: is it possible to change it cluster-template-ignition.yaml instead?

I think we should keep the core user there. It's sort of standard with Ignition I think? (IIRC)

Agree, core is the standard in ignition.

fabriziopandini · 2024-05-20T14:52:35Z

test/framework/log/collector.go

+			var errs []error
+			// Try with all available IPs unless it succeeded.
+			for _, machineIPAddress := range machineIPAddresses {
+				if err := executeRemoteCommand(f, machineIPAddress, command, args...); err != nil {


Q: does this imply we do expect ssh connectivity from the prow container to the machine? is this a new constraint introduced by this PR or it already exists for something else (thinking about constraints for the new test environment)

We already did this for a while to get logs

This PR just fixes the usage where we try to pull data via SSH to make it work in more failure cases. Before we tried but often did not get any data because there was no IP address yet set on the Machine's status object.

chrischdi · 2024-05-21T09:34:45Z

/test pull-cluster-api-provider-vsphere-e2e-govmomi-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-main

k8s-ci-robot · 2024-05-21T09:36:42Z

@chrischdi: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-vsphere-e2e-supervisor-main	`84beaf1`	link	true	`/test pull-cluster-api-provider-vsphere-e2e-supervisor-main`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

sbueringer · 2024-05-21T12:36:29Z

Merging, so we can rebase the bump PR. We can follow-up if necessary

/lgtm
/approve

k8s-ci-robot · 2024-05-21T12:36:34Z

LGTM label has been added.

Git tree hash: f5b6ed18d888b28a85a12c2c00e7918083e69758

k8s-ci-robot · 2024-05-21T12:36:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbueringer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sbueringer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-infra-cherrypick-robot · 2024-05-21T13:00:57Z

@chrischdi: #3010 failed to apply on top of branch "release-1.9":

Applying: test: fix collector for machines not having an IP in status
Using index info to reconstruct a base tree...
A	test/e2e/data/infrastructure-vsphere-govmomi/main/ignition/kustomization.yaml
M	test/e2e/e2e_suite_test.go
A	test/framework/log/collector.go
Falling back to patching base and 3-way merge...
Auto-merging test/e2e/log_collector.go
CONFLICT (content): Merge conflict in test/e2e/log_collector.go
Auto-merging test/e2e/e2e_suite_test.go
CONFLICT (content): Merge conflict in test/e2e/e2e_suite_test.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 test: fix collector for machines not having an IP in status
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-infra-cherrypick-robot · 2024-05-21T13:01:30Z

@chrischdi: #3010 failed to apply on top of branch "release-1.8":

Applying: test: fix collector for machines not having an IP in status
Using index info to reconstruct a base tree...
A	test/e2e/data/infrastructure-vsphere-govmomi/main/ignition/kustomization.yaml
M	test/e2e/e2e_suite_test.go
A	test/framework/log/collector.go
Falling back to patching base and 3-way merge...
Auto-merging test/e2e/log_collector.go
CONFLICT (content): Merge conflict in test/e2e/log_collector.go
Auto-merging test/e2e/e2e_suite_test.go
CONFLICT (content): Merge conflict in test/e2e/e2e_suite_test.go
Auto-merging test/e2e/data/infrastructure-vsphere/main/base/kustomization.yaml
CONFLICT (content): Merge conflict in test/e2e/data/infrastructure-vsphere/main/base/kustomization.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 test: fix collector for machines not having an IP in status
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.8

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-infra-cherrypick-robot · 2024-05-21T13:02:02Z

@chrischdi: #3010 failed to apply on top of branch "release-1.7":

Applying: test: fix collector for machines not having an IP in status
Using index info to reconstruct a base tree...
A	test/e2e/data/infrastructure-vsphere-govmomi/main/ignition/kustomization.yaml
M	test/e2e/e2e_suite_test.go
A	test/framework/log/collector.go
Falling back to patching base and 3-way merge...
Auto-merging test/e2e/log_collector.go
CONFLICT (content): Merge conflict in test/e2e/log_collector.go
Auto-merging test/e2e/e2e_suite_test.go
CONFLICT (content): Merge conflict in test/e2e/e2e_suite_test.go
Auto-merging test/e2e/data/infrastructure-vsphere/main/base/kustomization.yaml
CONFLICT (content): Merge conflict in test/e2e/data/infrastructure-vsphere/main/base/kustomization.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 test: fix collector for machines not having an IP in status
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-infra-cherrypick-robot · 2024-05-21T13:02:35Z

@chrischdi: #3010 failed to apply on top of branch "release-1.10":

Applying: test: fix collector for machines not having an IP in status
Applying: remove direct dependency on kind
Using index info to reconstruct a base tree...
M	Makefile
M	test/go.mod
Falling back to patching base and 3-way merge...
Auto-merging test/go.mod
CONFLICT (content): Merge conflict in test/go.mod
Auto-merging Makefile
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0002 remove direct dependency on kind
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 16, 2024

k8s-ci-robot requested review from MaxRink and vrabbi May 16, 2024 15:32

chrischdi changed the title ~~🌱 test: fix collector for machines not having an IP in status~~ 🌱 [WIP] test: fix collector for machines not having an IP in status May 16, 2024

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 16, 2024

chrischdi changed the title ~~🌱 [WIP] test: fix collector for machines not having an IP in status~~ 🌱 [WIP] test: fix collector for machines not having an IP in status and change ignition ssh user to capv May 16, 2024

chrischdi force-pushed the pr-fix-machine-collector branch from ac9992e to 0daf132 Compare May 16, 2024 16:42

chrischdi changed the title ~~🌱 [WIP] test: fix collector for machines not having an IP in status and change ignition ssh user to capv~~ 🌱 test: fix collector for machines not having an IP in status and change ignition ssh user to capv May 16, 2024

k8s-ci-robot assigned fabriziopandini and sbueringer May 17, 2024

sbueringer reviewed May 17, 2024

View reviewed changes

test/framework/log/collector.go Outdated Show resolved Hide resolved

test/framework/log/collector.go Outdated Show resolved Hide resolved

test/framework/log/collector.go Outdated Show resolved Hide resolved

test/framework/log/collector.go Outdated Show resolved Hide resolved

chrischdi force-pushed the pr-fix-machine-collector branch 2 times, most recently from 39b37f6 to a801c2a Compare May 17, 2024 12:12

test: fix collector for machines not having an IP in status

718cb44

chrischdi force-pushed the pr-fix-machine-collector branch from a801c2a to 718cb44 Compare May 17, 2024 12:15

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 17, 2024

fabriziopandini reviewed May 20, 2024

View reviewed changes

chrischdi mentioned this pull request May 21, 2024

🌱 Bump the all-go-mod-patch-and-minor group across 2 directories with 7 updates #3012

Closed

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 21, 2024

k8s-ci-robot requested review from fabriziopandini and sbueringer May 21, 2024 09:34

remove direct dependency on kind

279fddb

chrischdi force-pushed the pr-fix-machine-collector branch from 84beaf1 to 279fddb Compare May 21, 2024 09:39

chrischdi added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label May 21, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 21, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 21, 2024

k8s-ci-robot merged commit 172dc9b into kubernetes-sigs:main May 21, 2024
16 checks passed

k8s-ci-robot added this to the v1.11 milestone May 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🌱 test: fix collector for machines not having an IP in status and change ignition ssh user to capv #3010

🌱 test: fix collector for machines not having an IP in status and change ignition ssh user to capv #3010

chrischdi commented May 16, 2024 •

edited

Loading

chrischdi commented May 16, 2024

k8s-ci-robot commented May 16, 2024

chrischdi commented May 16, 2024

chrischdi commented May 17, 2024

k8s-infra-cherrypick-robot commented May 17, 2024

chrischdi commented May 17, 2024

k8s-infra-cherrypick-robot commented May 17, 2024

chrischdi commented May 17, 2024

k8s-infra-cherrypick-robot commented May 17, 2024

chrischdi commented May 17, 2024

k8s-infra-cherrypick-robot commented May 17, 2024

chrischdi commented May 17, 2024

sbueringer left a comment

sbueringer commented May 17, 2024

k8s-ci-robot commented May 17, 2024

fabriziopandini May 20, 2024

sbueringer May 21, 2024 •

edited

Loading

chrischdi May 21, 2024

fabriziopandini May 20, 2024

sbueringer May 21, 2024

chrischdi May 21, 2024

chrischdi commented May 21, 2024

k8s-ci-robot commented May 21, 2024 •

edited

Loading

sbueringer commented May 21, 2024

k8s-ci-robot commented May 21, 2024

k8s-ci-robot commented May 21, 2024

k8s-infra-cherrypick-robot commented May 21, 2024

k8s-infra-cherrypick-robot commented May 21, 2024

k8s-infra-cherrypick-robot commented May 21, 2024

k8s-infra-cherrypick-robot commented May 21, 2024

🌱 test: fix collector for machines not having an IP in status and change ignition ssh user to capv #3010

🌱 test: fix collector for machines not having an IP in status and change ignition ssh user to capv #3010

Conversation

chrischdi commented May 16, 2024 • edited Loading

chrischdi commented May 16, 2024

k8s-ci-robot commented May 16, 2024

chrischdi commented May 16, 2024

chrischdi commented May 17, 2024

k8s-infra-cherrypick-robot commented May 17, 2024

chrischdi commented May 17, 2024

k8s-infra-cherrypick-robot commented May 17, 2024

chrischdi commented May 17, 2024

k8s-infra-cherrypick-robot commented May 17, 2024

chrischdi commented May 17, 2024

k8s-infra-cherrypick-robot commented May 17, 2024

chrischdi commented May 17, 2024

sbueringer left a comment

Choose a reason for hiding this comment

sbueringer commented May 17, 2024

k8s-ci-robot commented May 17, 2024

fabriziopandini May 20, 2024

Choose a reason for hiding this comment

sbueringer May 21, 2024 • edited Loading

Choose a reason for hiding this comment

chrischdi May 21, 2024

Choose a reason for hiding this comment

fabriziopandini May 20, 2024

Choose a reason for hiding this comment

sbueringer May 21, 2024

Choose a reason for hiding this comment

chrischdi May 21, 2024

Choose a reason for hiding this comment

chrischdi commented May 21, 2024

k8s-ci-robot commented May 21, 2024 • edited Loading

sbueringer commented May 21, 2024

k8s-ci-robot commented May 21, 2024

k8s-ci-robot commented May 21, 2024

k8s-infra-cherrypick-robot commented May 21, 2024

k8s-infra-cherrypick-robot commented May 21, 2024

k8s-infra-cherrypick-robot commented May 21, 2024

k8s-infra-cherrypick-robot commented May 21, 2024

chrischdi commented May 16, 2024 •

edited

Loading

sbueringer May 21, 2024 •

edited

Loading

k8s-ci-robot commented May 21, 2024 •

edited

Loading