CAPVCD Control plane anti affinity rules

[vxav]

The control plane nodes should be spread on different vSphere hosts.

Filed upstream issue: vmware/cluster-api-provider-cloud-director#539

[vxav]

In a vSphere cluster, when an ESXi host fails, vSphere HA will restart the VMs running on it after a timeout.

To simulate a host failure where 2 CP nodes would be impacted, we can hard reboot 2 VMs at the same time to observe behaviour.

• After hard rebooting 2 CP nodes at the same time, the API becomes unavailable and comes back when the VMs restart.
• I also tried with all 3 control plane nodes and everything came back ok.

tl,dr: Having an anti-affinity rule would be nice as it would reduce downtime of the API in case of hardware/network failure, but we don't need to prioritise and take on the implementation of the feature.

@gawertm I propose that we move this issue to waiting in case a PE ever wants to take this on.

[vxav]

Given the recent news about the upstream project, I will close this issue as we won't implement this ourselves anyway.