[WIP] Increase parallelism for e2e tests

[shysank]

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

TODOs:

• squashed commits
• includes documentation
• adds unit tests

Release note:

none

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign cecilerobertmichon after the PR has been reviewed.
You can assign the PR to them by writing /assign @cecilerobertmichon in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

Two of the tests that failed in the last run didn't get reconciled at all in AzureClusterController and remains in provisioning state. This has been a somewhat consistent failure in most of the runs.

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

Run https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-azure/1816/pull-cluster-api-provider-azure-capi-e2e/1456472033914261504 passed in 1hr 29 min! Was it luck? Try again.

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

One more pass https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-azure/1816/pull-cluster-api-provider-azure-capi-e2e/1456648404498124800 in ~1hr30 mins. 2/3 passed. Cautiously optimistic. Try again.

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

uh oh it failed again https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-azure/1816/pull-cluster-api-provider-azure-capi-e2e/1456676924385398784. Again, not reconciling azure cluster objects. Perhaps the bootstrap cluster is overloaded with watches?

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

4 out of the last 5 tests passed. The one failure was due to a mhc flake. Looks like increasing the concurrency has resolved the AzureCluster reconciliation issue. Going to revert other hacks, and properly set concurrency in test, and try again for a few times.

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

Ginkgo node is not responding sometimes onsi/ginkgo#206 and times out after 4 hours. This is happening for quick start spec mostly. Going to add -debug and try again to find the cause as suggested in the issue.

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

Message: "admission webhook \"validation.kubeadmcontrolplane.controlplane.cluster.x-k8s.io\" denied the request: KubeadmControlPlane.controlplane.cluster.x-k8s.io \"kcp-upgrade-9hh6vc-control-plane\" is invalid: spec.kubeadmConfigSpec.clusterConfiguration.etcd.local.extraArgs: Forbidden: cannot be modified", This error is back again :(

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[shysank]

/test pull-cluster-api-provider-azure-capi-e2e

[k8s-ci-robot]

@shysank: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-azure-e2e-exp	2ff77ac	link	false	/test pull-cluster-api-provider-azure-e2e-exp
pull-cluster-api-provider-azure-capi-e2e	2ff77ac	link	false	/test pull-cluster-api-provider-azure-capi-e2e

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[CecileRobertMichon]

I think we can close this now that we somewhat solved our problem by removing upgrade from spec

Still something that might be worth revisiting later but for now

/close

[k8s-ci-robot]

@CecileRobertMichon: Closed this PR.

In response to this:

I think we can close this now that we somewhat solved our problem by removing upgrade from spec

Still something that might be worth revisiting later but for now

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.