using azurefile-csi-driver in k3d

[bartr]

I'm following the instructions here: https://github.com/kubernetes-sigs/azurefile-csi-driver/blob/master/deploy/example/e2e_usage.md#option2-pvpvc

I have a clean k3d cluster and I'm able to mount my azure file share from the host and the node. Should this work? I've tried all of the different methods on that page and get the same error.

Everything looks good but when I deploy the nginx app (or any app) that uses the PVC, the pod hangs in creating. After a couple of minutes, I get the following errrors:

Type     Reason              Age   From                     Message
  ----     ------              ----  ----                     -------
  Normal   Scheduled           3m5s  default-scheduler        Successfully assigned default/nginx-azurefile to k3d-k3s-default-server-0
  Warning  FailedMount         63s   kubelet                  Unable to attach or mount volumes: unmounted volumes=[persistent-storage], unattached volumes=[persistent-storage kube-api-access-4pdh5]: timed out waiting for the condition
  Warning  FailedAttachVolume  27s   attachdetach-controller  AttachVolume.Attach failed for volume "pv-azurefile" : Attach timeout for volume upload-volumeid

Here's my PV/PVC yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-azurefile
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: azurefile-csi
  mountOptions:
    - dir_mode=0777
    - file_mode=0777
    - uid=0
    - gid=0
    - mfsymlinks
    - cache=strict  # https://linux.die.net/man/8/mount.cifs
    - nosharesock
  csi:
    driver: file.csi.azure.com
    readOnly: false
    # make sure this volumeid is unique in the cluster
    # `#` is not allowed in self defined volumeHandle
    volumeHandle: upload-volumeid
    volumeAttributes:
      shareName: uploadvolume
    nodeStageSecretRef:
      name: azure-secret
      namespace: default
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-azurefile
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: azurefile-csi

k describe pvc pvc-azure output

Name:          pvc-azurefile
Namespace:     default
StorageClass:  azurefile-csi
Status:        Bound
Volume:        pv-azurefile
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      10Gi
Access Modes:  RWX
VolumeMode:    Filesystem
Used By:       nginx-azurefile
Events:        <none>

[andyzhangx]

what's the k8s version? not sure why there is attachdetach-controller AttachVolume.Attach process, and what's the csi driver version you have deployed? With latest release version(v1.16.0), there should be only kubelet MountVolume.MountDevice events directly using PV/PVC way.

[bartr]

Image: mcr.microsoft.com/k8s/csi/azurefile-csi:latest

I got an error in the node that I hadn't noticed before:

  Normal   Started    5m45s  kubelet            Started container node-driver-registrar
  Normal   Pulling    5m45s  kubelet            Pulling image "mcr.microsoft.com/k8s/csi/azurefile-csi:latest"
  Normal   Pulled     5m41s  kubelet            Successfully pulled image "mcr.microsoft.com/k8s/csi/azurefile-csi:latest" in 4.523367931s
  Warning  Failed     5m41s  kubelet            Error: failed to generate container "ec48be6e8bd8a4043b27c6747a09524d035fe05cbcbbc5f1eb42ba70c2dd86b2" spec: failed to generate spec: path "/var/lib/kubelet/" is mounted on "/var/lib/kubelet" but it is not a shared mount

[andyzhangx]

follow this guide to set kubelet path: kubernetes-csi/csi-driver-smb#335 (comment)
and also pls don't use latest tag which is only for testing or trying new features

[bartr]

Setting /var/lib/kubelet to shared worked.

Here's what I did for k3d

sudo mkdir -p /k3d/var/lib/kubelet
sudo chown -R $USER:$USER /k3d
sudo mount -o bind /k3d/var/lib/kubelet /k3d/var/lib/kubelet
sudo mount --make-shared /k3d/var/lib/kubelet

k3d cluster create ... \
-v /k3d/var/lib/kubelet:/var/lib/kubelet:shared

I also created my azure.json file and mounted it

sudo mkdir -p /k3d/etc/kubernetes

# echo to /k3d/etc/kubernetes/azure.json

k3d cluster create ... \
-v /k3d/var/lib/kubelet:/var/lib/kubelet:shared \
-v /k3d/etc/kubernetes:/etc/kubernetes

[andyzhangx]

so azure file driver works on your cluster now? what's the original issue? caused by latest tag?

[bartr]

the issue was /var/lib/kubelet not being a mount or shared.