VMware Cloud Director Machines are not getting provisioned if a SSH key is provided

[xmudrii]

If you create a VMware Cloud Director (VCD) based Machine with a SSH key, the Machine/VM never gets fully-provisioned. The VM is created, it has an IP address assigned, but it doesn't join the cluster. These are the findings after some initial debugging:

• The SSH key is picked up and it's possible to access the Machine via SSH
• There are some problems with cloud-init and the bootstrap service that we run as part of OSM (which can be seen from the logs)

Removing the SSH key from the Machine mitigates the issue. In this case, the VM gets fully provisioned and it joins the cluster.

Some additional breadcrumbs for debugging this:

• We should check the OVF property vcloudapp.enable_ssh.VMware_vCloud_Director, if you have this enabled, whatever you add to the OS will be overwritten
• This was tested on Ubuntu, we should also test Flatcar to check if it's affected
• This is affecting both KKP and KubeOne, this issue might span across VCD, machine-controller and operating-system-manager

[ahmedwaleedmalik]

For whoever works on this, you'll be able to find more info regarding this topic in this internal thread(private).