Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🌱 [WIP] test: stream logs for CPI and CSI to artifacts #2615

Closed
wants to merge 2 commits into from
Closed

🌱 [WIP] test: stream logs for CPI and CSI to artifacts #2615

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
89dc2ed
test: stream logs for CPI and CSI to artifacts
chrischdi Jan 15, 2024
2698764
fix
chrischdi Jan 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions packaging/flavorgen/cloudprovider/csi/vsphere-csi-driver.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,8 @@ apiVersion: apps/v1
metadata:
name: vsphere-csi-controller
namespace: vmware-system-csi
labels:
component: "vsphere-csi"
spec:
replicas: 3
strategy:
Expand Down Expand Up @@ -579,6 +581,8 @@ apiVersion: apps/v1
metadata:
name: vsphere-csi-node-windows
namespace: vmware-system-csi
labels:
component: "vsphere-csi"
spec:
selector:
matchLabels:
Expand Down
4 changes: 4 additions & 0 deletions templates/cluster-template-external-loadbalancer.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -588,6 +588,8 @@ data:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
component: vsphere-csi
name: vsphere-csi-controller
namespace: vmware-system-csi
spec:
Expand Down Expand Up @@ -957,6 +959,8 @@ data:
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
component: vsphere-csi
name: vsphere-csi-node-windows
namespace: vmware-system-csi
spec:
Expand Down
4 changes: 4 additions & 0 deletions templates/cluster-template-ignition.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -792,6 +792,8 @@ data:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
component: vsphere-csi
name: vsphere-csi-controller
namespace: vmware-system-csi
spec:
Expand Down Expand Up @@ -1161,6 +1163,8 @@ data:
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
component: vsphere-csi
name: vsphere-csi-node-windows
namespace: vmware-system-csi
spec:
Expand Down
4 changes: 4 additions & 0 deletions templates/cluster-template-node-ipam.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -713,6 +713,8 @@ data:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
component: vsphere-csi
name: vsphere-csi-controller
namespace: vmware-system-csi
spec:
Expand Down Expand Up @@ -1082,6 +1084,8 @@ data:
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
component: vsphere-csi
name: vsphere-csi-node-windows
namespace: vmware-system-csi
spec:
Expand Down
4 changes: 4 additions & 0 deletions templates/cluster-template-topology.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -487,6 +487,8 @@ data:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
component: vsphere-csi
name: vsphere-csi-controller
namespace: vmware-system-csi
spec:
Expand Down Expand Up @@ -856,6 +858,8 @@ data:
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
component: vsphere-csi
name: vsphere-csi-node-windows
namespace: vmware-system-csi
spec:
Expand Down
4 changes: 4 additions & 0 deletions templates/cluster-template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -703,6 +703,8 @@ data:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
component: vsphere-csi
name: vsphere-csi-controller
namespace: vmware-system-csi
spec:
Expand Down Expand Up @@ -1072,6 +1074,8 @@ data:
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
component: vsphere-csi
name: vsphere-csi-node-windows
namespace: vmware-system-csi
spec:
Expand Down
3 changes: 3 additions & 0 deletions test/e2e/anti_affinity_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,9 @@ func VerifyAntiAffinity(ctx context.Context, input AntiAffinitySpecInput) {
WaitForClusterIntervals: input.Global.E2EConfig.GetIntervals("", "wait-cluster"),
WaitForControlPlaneIntervals: input.Global.E2EConfig.GetIntervals("", "wait-control-plane"),
WaitForMachineDeployments: input.Global.E2EConfig.GetIntervals("", "wait-worker-nodes"),
PostMachinesProvisioned: func() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be PostClusterInitialized?
(with some tolerance to give time to CRS to be applied, assuming we are using CRS to deploy the CSI/CPI)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was not able to find a PostClusterInitialized 🤔

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: have to rethink this, PostMachinesProvisioned won't be very helpful because this is more on a happy path then. Machines may not even get provisioned and we want the logs.

watchVSphereComponentLogs(ctx, artifactFolder, input.Global.BootstrapClusterProxy, namespace.Name, clusterName)
},
}, clusterResources)

vsphereCluster := FetchVSphereClusterObject(ctx, input.Global.BootstrapClusterProxy, client.ObjectKey{
Expand Down
4 changes: 3 additions & 1 deletion test/e2e/dhcp_overrides_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,9 @@ var _ = Describe("DHCPOverrides configuration test", func() {
WaitForClusterIntervals: e2eConfig.GetIntervals(specName, "wait-cluster"),
WaitForControlPlaneIntervals: e2eConfig.GetIntervals(specName, "wait-control-plane"),
WaitForMachineDeployments: e2eConfig.GetIntervals(specName, "wait-worker-nodes"),
}, clusterResources)
PostMachinesProvisioned: func() {
watchVSphereComponentLogs(ctx, artifactFolder, bootstrapClusterProxy, namespace.Name, clusterName)
}}, clusterResources)

list := getVSphereVMsForCluster(clusterName, namespace.Name)
for _, vm := range list.Items {
Expand Down
3 changes: 3 additions & 0 deletions test/e2e/gpu_pci_passthrough_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,9 @@ var _ = Describe("Cluster creation with GPU devices as PCI passthrough [speciali
WaitForClusterIntervals: e2eConfig.GetIntervals("", "wait-cluster"),
WaitForControlPlaneIntervals: e2eConfig.GetIntervals("", "wait-control-plane"),
WaitForMachineDeployments: e2eConfig.GetIntervals("", "wait-worker-nodes"),
PostMachinesProvisioned: func() {
watchVSphereComponentLogs(ctx, artifactFolder, bootstrapClusterProxy, namespace.Name, clusterName)
},
}, &clusterctl.ApplyClusterTemplateAndWaitResult{})

By("Verifying that the PCI device is attached to the worker node")
Expand Down
3 changes: 3 additions & 0 deletions test/e2e/hardware_upgrade_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,9 @@ func VerifyHardwareUpgrade(ctx context.Context, input HardwareUpgradeSpecInput)
WaitForClusterIntervals: input.Global.E2EConfig.GetIntervals(specName, "wait-cluster"),
WaitForControlPlaneIntervals: input.Global.E2EConfig.GetIntervals(specName, "wait-control-plane"),
WaitForMachineDeployments: input.Global.E2EConfig.GetIntervals(specName, "wait-worker-nodes"),
PostMachinesProvisioned: func() {
watchVSphereComponentLogs(ctx, artifactFolder, bootstrapClusterProxy, namespace.Name, clusterName)
},
}, clusterResources)

Byf("Fetching the VSphereVM objects for the cluster %s", clusterName)
Expand Down
43 changes: 43 additions & 0 deletions test/e2e/log_collector.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ import (
"golang.org/x/crypto/ssh"
clusterv1 "sigs.k8s.io/cluster-api/api/v1beta1"
expv1 "sigs.k8s.io/cluster-api/exp/api/v1beta1"
"sigs.k8s.io/cluster-api/test/framework"
"sigs.k8s.io/controller-runtime/pkg/client"
kinderrors "sigs.k8s.io/kind/pkg/errors"
)
Expand Down Expand Up @@ -154,3 +155,45 @@ func readPrivateKey() ([]byte, error) {

return os.ReadFile(filepath.Clean(privateKeyFilePath))
}

func watchVSphereComponentLogs(ctx context.Context, artifactFolder string, managementClusterProxy framework.ClusterProxy, workloadClusterNamespace, workloadClusterName string) {
workloadClusterProxy := managementClusterProxy.GetWorkloadCluster(ctx, workloadClusterNamespace, workloadClusterName)
// CPI
framework.WatchDaemonSetLogsByLabelSelector(ctx, framework.WatchDaemonSetLogsByLabelSelectorInput{
GetLister: workloadClusterProxy.GetClient(),
Cache: workloadClusterProxy.GetCache(ctx),
ClientSet: workloadClusterProxy.GetClientSet(),
Labels: map[string]string{
"component": "cloud-controller-manager",
},
LogPath: filepath.Join(artifactFolder, "clusters", workloadClusterName, "logs"),
})

// CSI Deployment
framework.WatchDeploymentLogsByLabelSelector(ctx, framework.WatchDeploymentLogsByLabelSelectorInput{
GetLister: workloadClusterProxy.GetClient(),
Cache: workloadClusterProxy.GetCache(ctx),
ClientSet: workloadClusterProxy.GetClientSet(),
Labels: map[string]string{
"component": "vsphere-csi",
},
LogPath: filepath.Join(artifactFolder, "clusters", workloadClusterName, "logs"),
})

// CSI Daemonset
framework.WatchDaemonSetLogsByLabelSelector(ctx, framework.WatchDaemonSetLogsByLabelSelectorInput{
GetLister: workloadClusterProxy.GetClient(),
Cache: workloadClusterProxy.GetCache(ctx),
ClientSet: workloadClusterProxy.GetClientSet(),
Labels: map[string]string{
"component": "vsphere-csi",
},
LogPath: filepath.Join(artifactFolder, "clusters", workloadClusterName, "logs"),
})
}

func watchVSphereComponentLogsFunc(ctx context.Context, artifactFolder string) func(managementClusterProxy framework.ClusterProxy, workloadClusterNamespace, workloadClusterName string) {
return func(managementClusterProxy framework.ClusterProxy, workloadClusterNamespace, workloadClusterName string) {
watchVSphereComponentLogs(ctx, artifactFolder, managementClusterProxy, workloadClusterNamespace, workloadClusterName)
}
}
3 changes: 3 additions & 0 deletions test/e2e/node_labeling_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,9 @@ func VerifyNodeLabeling(ctx context.Context, input NodeLabelingSpecInput) {
WaitForClusterIntervals: input.Global.E2EConfig.GetIntervals("", "wait-cluster"),
WaitForControlPlaneIntervals: input.Global.E2EConfig.GetIntervals("", "wait-control-plane"),
WaitForMachineDeployments: input.Global.E2EConfig.GetIntervals("", "wait-worker-nodes"),
PostMachinesProvisioned: func() {
watchVSphereComponentLogs(ctx, artifactFolder, input.Global.BootstrapClusterProxy, namespace.Name, clusterName)
},
}, clusterResources)
workloadProxy := input.Global.BootstrapClusterProxy.GetWorkloadCluster(ctx, namespace.Name, clusterResources.Cluster.Name)

Expand Down
2 changes: 2 additions & 0 deletions test/e2e/ownerreference_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,8 @@ var _ = Describe("OwnerReference checks with FailureDomains and ClusterIdentity"
VSphereKubernetesReferenceAssertions,
VSphereReferenceAssertions,
)

watchVSphereComponentLogs(ctx, artifactFolder, bootstrapClusterProxy, namespace, clusterName)
},
}
})
Expand Down
26 changes: 14 additions & 12 deletions test/e2e/quick_start_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,25 +37,27 @@ var _ = Describe("Cluster Creation using Cluster API quick-start test", func() {
var _ = Describe("ClusterClass Creation using Cluster API quick-start test [PR-Blocking] [ClusterClass]", func() {
capi_e2e.QuickStartSpec(ctx, func() capi_e2e.QuickStartSpecInput {
return capi_e2e.QuickStartSpecInput{
E2EConfig: e2eConfig,
ClusterctlConfigPath: clusterctlConfigPath,
BootstrapClusterProxy: bootstrapClusterProxy,
ArtifactFolder: artifactFolder,
SkipCleanup: skipCleanup,
Flavor: ptr.To("topology"),
E2EConfig: e2eConfig,
ClusterctlConfigPath: clusterctlConfigPath,
BootstrapClusterProxy: bootstrapClusterProxy,
ArtifactFolder: artifactFolder,
SkipCleanup: skipCleanup,
Flavor: ptr.To("topology"),
PostMachinesProvisioned: watchVSphereComponentLogsFunc(ctx, artifactFolder),
}
})
})

var _ = Describe("Cluster creation with [Ignition] bootstrap [PR-Blocking]", func() {
capi_e2e.QuickStartSpec(ctx, func() capi_e2e.QuickStartSpecInput {
return capi_e2e.QuickStartSpecInput{
E2EConfig: e2eConfig,
ClusterctlConfigPath: clusterctlConfigPath,
BootstrapClusterProxy: bootstrapClusterProxy,
ArtifactFolder: artifactFolder,
SkipCleanup: skipCleanup,
Flavor: ptr.To("ignition"),
E2EConfig: e2eConfig,
ClusterctlConfigPath: clusterctlConfigPath,
BootstrapClusterProxy: bootstrapClusterProxy,
ArtifactFolder: artifactFolder,
SkipCleanup: skipCleanup,
Flavor: ptr.To("ignition"),
PostMachinesProvisioned: watchVSphereComponentLogsFunc(ctx, artifactFolder),
}
})
})
5 changes: 4 additions & 1 deletion test/e2e/storage_policy_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ func VerifyStoragePolicy(ctx context.Context, input StoragePolicySpecInput) {

By("creating a workload cluster")
configCluster := defaultConfigCluster(clusterName, namespace.Name, specName, 1, 0, GlobalInput{
BootstrapClusterProxy: bootstrapClusterProxy,
BootstrapClusterProxy: input.Global.BootstrapClusterProxy,
ClusterctlConfigPath: clusterctlConfigPath,
E2EConfig: e2eConfig,
ArtifactFolder: artifactFolder,
Expand All @@ -97,6 +97,9 @@ func VerifyStoragePolicy(ctx context.Context, input StoragePolicySpecInput) {
WaitForClusterIntervals: input.Global.E2EConfig.GetIntervals("", "wait-cluster"),
WaitForControlPlaneIntervals: input.Global.E2EConfig.GetIntervals("", "wait-control-plane"),
WaitForMachineDeployments: input.Global.E2EConfig.GetIntervals("", "wait-worker-nodes"),
PostMachinesProvisioned: func() {
watchVSphereComponentLogs(ctx, artifactFolder, input.Global.BootstrapClusterProxy, namespace.Name, clusterName)
},
}, clusterResources)

pbmClient, err := pbm.NewClient(ctx, input.Client.Client)
Expand Down
2 changes: 1 addition & 1 deletion test/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ go 1.20

replace sigs.k8s.io/cluster-api => sigs.k8s.io/cluster-api v1.6.0

replace sigs.k8s.io/cluster-api/test => sigs.k8s.io/cluster-api/test v1.6.1-0.20240108181656-eb46d086f417
replace sigs.k8s.io/cluster-api/test => sigs.k8s.io/cluster-api/test v1.6.1-0.20240112150242-70716332b487

replace sigs.k8s.io/cluster-api-provider-vsphere => ../

Expand Down
4 changes: 2 additions & 2 deletions test/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1115,8 +1115,8 @@ rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=
sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.0.15/go.mod h1:LEScyzhFmoF5pso/YSeBstl57mOzx9xlU9n85RGrDQg=
sigs.k8s.io/cluster-api v1.6.0 h1:2bhVSnUbtWI8taCjd9lGiHExsRUpKf7Z1fXqi/IwYx4=
sigs.k8s.io/cluster-api v1.6.0/go.mod h1:LB7u/WxiWj4/bbpHNOa1oQ8nq0MQ5iYlD0pGfRSBGLI=
sigs.k8s.io/cluster-api/test v1.6.1-0.20240108181656-eb46d086f417 h1:aI5QL5q0eT20HyA36LQfpkTVu7jdM9ssmNzBnA+gRpo=
sigs.k8s.io/cluster-api/test v1.6.1-0.20240108181656-eb46d086f417/go.mod h1:hsaaOsAftllhnRZ6l6izUvgGswlqO8Gm4EQS96+mnC0=
sigs.k8s.io/cluster-api/test v1.6.1-0.20240112150242-70716332b487 h1:72l+q8XVnAtW74R0slvFDljpjjtK+PJwZYSPphTlNRg=
sigs.k8s.io/cluster-api/test v1.6.1-0.20240112150242-70716332b487/go.mod h1:hsaaOsAftllhnRZ6l6izUvgGswlqO8Gm4EQS96+mnC0=
sigs.k8s.io/controller-runtime v0.9.0/go.mod h1:TgkfvrhhEw3PlI0BRL/5xM+89y3/yc0ZDfdbTl84si8=
sigs.k8s.io/controller-runtime v0.16.3 h1:2TuvuokmfXvDUamSx1SuAOO3eTyye+47mJCigwG62c4=
sigs.k8s.io/controller-runtime v0.16.3/go.mod h1:j7bialYoSn142nv9sCOJmQgDXQXxnroFU4VnX/brVJ0=
Expand Down
Loading