Invalid configuration for device '0'. - After migrating VMs to a different datastore

[benedikt-haug]

Dear developers,

we encountered an issue:
We migrated the storage vmotion of 3 datastores. Afterwards the PVC couldn't be mounted and the vsphere client constantly shows the kubernetes machines fail to configure the disks with the error : "Invalid configuration for device '0'".

What we found interesting was that the kubevols were not migrated along with the VMDKs. We tried to move them to the VM's datastore manually but that didn't change anything. Rebooted the VMs a few times.

Attached logs I thought may be relevant. Please get in touch if you want to see something additional.

Thanks a lot!

kube-controller-manager:

    E0914 13:25:46.448746       1 vsphere.go:929] Failed to attach disk: [ds-pod0304-cage-12] kubevols/kubernetes-dynamic-pvc-a15dc739-d2a3-48e7-b37a-d61d415f85f0.vmdk for node: xtc651. err: +Invalid configuration for device '0'.
    E0914 13:25:46.448757       1 attacher.go:93] Error attaching volume "[ds-pod0304-cage-12] kubevols/kubernetes-dynamic-pvc-a15dc739-d2a3-48e7-b37a-d61d415f85f0.vmdk" to node "xtc651": Invalid configuration for device '0'.
    I0914 13:25:46.448779       1 actual_state_of_world.go:350] Volume "kubernetes.io/vsphere-volume/[ds-pod0304-cage-12] kubevols/kubernetes-dynamic-pvc-a15dc739-d2a3-48e7-b37a-d61d415f85f0.vmdk" is already added to attachedVolume list to node "xtc651", update device path ""
    E0914 13:25:46.448827       1 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/vsphere-volume/[ds-pod0304-cage-12] kubevols/kubernetes-dynamic-pvc-a15dc739-d2a3-48e7-b37a-d61d415f85f0.vmdk podName: nodeName:}" failed. No retries permitted until 2021-09-14 13:27:48.448809285 +0000 UTC m=+19048.090430495 (durationBeforeRetry 2m2s). Error: "AttachVolume.Attach failed for volume \"pvc-a15dc739-d2a3-48e7-b37a-d61d415f85f0\" (UniqueName: \"kubernetes.io/vsphere-volume/[ds-pod0304-cage-12] kubevols/kubernetes-dynamic-pvc-a15dc739-d2a3-48e7-b37a-d61d415f85f0.vmdk\") from node \"xtc651\" : Invalid configuration for device '0'."
    I0914 13:25:46.448851       1 event.go:291] "Event occurred" object="prometheus-operator/alertmanager-kube-prometheus-stack-t-alertmanager-0" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"pvc-a15dc739-d2a3-48e7-b37a-d61d415f85f0\" : Invalid configuration for device '0'."

storageprovider:

    Provisioner:           kubernetes.io/vsphere-volume
    Parameters:            <none>
    AllowVolumeExpansion:  <unset>
    MountOptions:          <none>
    ReclaimPolicy:         Delete
    VolumeBindingMode:     Immediate
    Events:                <none>

pvc:

    Name:          alertmanager-kube-prometheus-stack-t-alertmanager-db-alertmanager-kube-prometheus-stack-t-alertmanager-0
    Namespace:     prometheus-operator
    StorageClass:  vsphere
    Status:        Bound
    Volume:        pvc-a15dc739-d2a3-48e7-b37a-d61d415f85f0
    Labels:        alertmanager=kube-prometheus-stack-t-alertmanager
                   app=alertmanager
    Annotations:   pv.kubernetes.io/bind-completed: yes
                   pv.kubernetes.io/bound-by-controller: yes
                   volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume
    Finalizers:    [kubernetes.io/pvc-protection]
    Capacity:      25Gi
    Access Modes:  RWO
    VolumeMode:    Filesystem
    Mounted By:    alertmanager-kube-prometheus-stack-t-alertmanager-0
    Events:        <none>

pv:

    Name:            pvc-a15dc739-d2a3-48e7-b37a-d61d415f85f0
    Labels:          <none>
    Annotations:     kubernetes.io/createdby: vsphere-volume-dynamic-provisioner
                     pv.kubernetes.io/bound-by-controller: yes
                     pv.kubernetes.io/provisioned-by: kubernetes.io/vsphere-volume
    Finalizers:      [kubernetes.io/pv-protection]
    StorageClass:    vsphere
    Status:          Bound
    Claim:           prometheus-operator/alertmanager-kube-prometheus-stack-t-alertmanager-db-alertmanager-kube-prometheus-stack-t-alertmanager-0
    Reclaim Policy:  Delete
    Access Modes:    RWO
    VolumeMode:      Filesystem
    Capacity:        25Gi
    Node Affinity:   <none>
    Message:
    Source:
        Type:               vSphereVolume (a Persistent Disk resource in vSphere)
        VolumePath:         [ds-pod0304-cage-12] kubevols/kubernetes-dynamic-pvc-a15dc739-d2a3-48e7-b37a-d61d415f85f0.vmdk
        FSType:             ext4
        StoragePolicyName:
    Events:                 <none>