🐛 StoragePolicy should select datastore based on connected hosts and set on VM spec

[rikatz]

What this PR does / why we need it:
This PR fixes some workflows on CAPV + Storage policy:

• When using a storage policy, we need to make sure that the only selected datastores are the DS connected to the cluster/host. Today we don't validate it, which means that we may end up with an invalid datastore on a multi-cluster environment with a storage policy that selects datastores on different clusters, we can select a datastore not connected to our cluster - The new workflow will first get all the datastores connected to the cluster, and then pass this list of datastores to Storage Policy validation
• When passing both storage policy and datastore, this datastore should be validated on the storage policy and fail if it cannot be used
• Finally, on cloning we MUST pass spec.Location.Datastore = datastoreRef otherwise the cloning process will not understand when this is a cross datastore/cluster cloning and will fail, as it will try to use the template datastore.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

[randomvariable]

/hold

@rikatz is testing on linked clone workflow, but otherwise ready for review.

[rikatz]

@randomvariable @sbueringer just confirming that the linked clone still works with this change. I will unhold, but feel free to merge it after the next minor release, as a patch if the time line needs to! :)

/hold cancel

[chrischdi]

/test help

[k8s-ci-robot]

@chrischdi: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

• /test pull-cluster-api-provider-vsphere-conformance-main
• /test pull-cluster-api-provider-vsphere-e2e-full-main
• /test pull-cluster-api-provider-vsphere-e2e-main
• /test pull-cluster-api-provider-vsphere-test-integration-main
• /test pull-cluster-api-provider-vsphere-test-main
• /test pull-cluster-api-provider-vsphere-verify-main

The following commands are available to trigger optional jobs:

• /test pull-cluster-api-provider-vsphere-apidiff-main

Use /test all to run the following jobs that were automatically triggered:

• pull-cluster-api-provider-vsphere-apidiff-main
• pull-cluster-api-provider-vsphere-e2e-main
• pull-cluster-api-provider-vsphere-test-integration-main
• pull-cluster-api-provider-vsphere-test-main
• pull-cluster-api-provider-vsphere-verify-main

In response to this:

/test help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[chrischdi]

/test pull-cluster-api-provider-vsphere-e2e-full-main

[chrischdi]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: f0fa4d8606ab15ac9c20e42107a8a93a94228115

[sbueringer]

Overall makes sense to me.

Some suggestions for godoc.

I would like a review from either @srm09 or @yastij to ensure we're not missing anything

/assign @srm09 @yastij

[sbueringer]

@randomvariable If you have some time, would be good if you can take a look

[rikatz]

/label tide/merge-method-squash

[sbueringer]

Perfect. Thank you very much!

/lgtm
/approve

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: a04ef759d82e5f997cda590c0e17c4e297ff5685

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbueringer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [sbueringer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sbueringer]

/cherry-pick release-1.8

[k8s-infra-cherrypick-robot]

@sbueringer: once the present PR merges, I will cherry-pick it on top of release-1.8 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.8

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sbueringer]

/cherry-pick release-1.7

[k8s-infra-cherrypick-robot]

@sbueringer: once the present PR merges, I will cherry-pick it on top of release-1.7 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sbueringer]

/cherry-pick release-1.6

[k8s-infra-cherrypick-robot]

@sbueringer: once the present PR merges, I will cherry-pick it on top of release-1.6 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sbueringer]

/cherry-pick release-1.5

[k8s-infra-cherrypick-robot]

@sbueringer: once the present PR merges, I will cherry-pick it on top of release-1.5 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-infra-cherrypick-robot]

@sbueringer: new pull request created: #2275

In response to this:

/cherry-pick release-1.8

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-infra-cherrypick-robot]

@sbueringer: #2134 failed to apply on top of branch "release-1.7":

Applying: Storagepolicy should select ds based on connected cluster
Using index info to reconstruct a base tree...
M	pkg/services/govmomi/vcenter/clone.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/services/govmomi/vcenter/clone.go
CONFLICT (content): Merge conflict in pkg/services/govmomi/vcenter/clone.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Storagepolicy should select ds based on connected cluster
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-infra-cherrypick-robot]

@sbueringer: #2134 failed to apply on top of branch "release-1.6":

Applying: Storagepolicy should select ds based on connected cluster
Using index info to reconstruct a base tree...
M	pkg/services/govmomi/vcenter/clone.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/services/govmomi/vcenter/clone.go
CONFLICT (content): Merge conflict in pkg/services/govmomi/vcenter/clone.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Storagepolicy should select ds based on connected cluster
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-infra-cherrypick-robot]

@sbueringer: #2134 failed to apply on top of branch "release-1.5":

Applying: Storagepolicy should select ds based on connected cluster
Using index info to reconstruct a base tree...
M	pkg/services/govmomi/vcenter/clone.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/services/govmomi/vcenter/clone.go
CONFLICT (content): Merge conflict in pkg/services/govmomi/vcenter/clone.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Storagepolicy should select ds based on connected cluster
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sbueringer]

@rikatz Do you have some time to open cherry-picks for the other branches? I diffed the branches and thought they are all pretty similar :/

[rikatz]

done for the 3 failed cherry picks