ESXi 6.5 60 volume limit per cluster

[compendius]

Individual VMs in VMware have a limitation of 60 virtual disks per VM in ESXi 6.5

The Kubernetes scheduler is not aware of this limitation, and does not account for the amount of virtual disks already attached to a VM when pods with dynamic persistent vSphere storage class volumes are provisioned and therefore add more virtual disks.

Therefore no matter how large a Kubernetes cluster you can only safely ever use 60 vSphere storage class volumes across the whole cluster. If the scheduler picks a node that already has 60 virtual disks then the deployment will fail and not try reschedule onto a different node.

This maximum scale limit documentation is misleading in this and makes no mention of the limitation.

https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/maximum-scale-limit.html

[compendius]

After more research it looks like Kubernetes 1.11+ has a feature called 'node-specific volume limits"

https://kubernetes.io/docs/concepts/storage/storage-limits/

To fix this issue the vSphere driver needs to support this feature. It can do this either with a re-write of the in-tree driver, or implementing CSI as the storage interface (which will support this).

[divyenpatel]

@compendius thank you for reporting an issue.
Can you file a feature request at - https://github.com/kubernetes/cloud-provider-vsphere

We will implement this with CSI.

CC: @SandeepPissay