Pods having permissions issue with NFS share

[michae1ho11ey]

To start out with I'd like to say I'm a sys admin over his head with a requirement to stand up an Kubernetes cluster. We don't have a Kubernetes team or an expert, just me trying to figure things out based on documentation. I'm at a point that I'm not sure what else to try. I apologize if this is in the wrong place.

I recently stood up a vanilla Kubernetes cluster using kubeadm on Oracle Linux hosts. This cluster is to host a VMWare NSX monitoring suite of tools. Once the cluster was stood up, we inputed some details about the cluster, gave the installer the KubeConfig file and let it go to town. After several bumps, I think we've hit the last one, the pods have access to the NFS server but it's having file permissions with the shares and I don't know if that is a Kuberenetes service account/RBAC issue, I failed to configure the storage class correctly, or something else. I'd appreciate anyone's help and pointing me in the right direction with this.

Here are two errors I'm seeing from pods that has lead me to thing I'm having NFS permissions issues.
Error: Drive https://minio-3.minio-headless.nsxi-platform.svc.cluster.local:9000/data/minio, path (.minio.sys/pool.bin) returned an error (file access denied) (*fmt.wrapError)

and

[164] FATAL: data directory "/bitnami/postgresql/data" has wrong ownership

• The cluster was originally set up as 1.21 but I've since upgraded to 1.24.9.
• I have csi-driver-nfs version v4.1.0
• It's connecting to a distributed file system NFS share via a URL. I had to use the options nfsvers=4.2,sec=none to get the pods to successfully connect.
• Using the above mentioned options, I'm able to connect to the NFS share from all the host machines.
• After the the pods gained access via the NFS storage class, they do have write permissions as I can see files in some of the directories created.
• I can see the UID and GUID of each of the shares (folders under /kubernetes) as 4294967294 with 777 permissions.
• Sub directories under a single share still have the same UID:GUID but now the files have 755 permissions.
• Here is the storage class definition:

---

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-csi
provisioner: nfs.csi.k8s.io
parameters:
server: infra-files.mycompany.com
share: /kubernetes
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:

• nfsvers=4.2
• sec=none

I manually mounted the NFS share to view permissions, here are the excerpts:

$ ls -al /mnt
drwxrwxrwt. 45 root root 45 Jan 6 18:44 .
dr-xr-xr-x. 17 root root 224 Apr 29 2022 ..
drwxrwsrwx. 4 4294967294 4294967294 4 Jan 6 19:26 pvc-02cb3ac6-1ef9-4bbb-b8bc-c90a5d6ce846

Going one folder down:

$ ls -al /mnt/pvc-02cb3ac6-1ef9-4bbb-b8bc-c90a5d6ce846
drwxrwsrwx. 5 4294967294 4294967294 5 Jan 6 18:43 .
drwxrwxrwt. 45 root root 45 Jan 6 18:44 ..
drwxr-sr-x. 3 4294967294 4294967294 3 Jan 6 18:44 druid
drwxr-sr-x. 2 4294967294 4294967294 2 Jan 6 18:43 dump
drwxr-sr-x. 2 4294967294 4294967294 3 Jan 6 18:43 tmp

[andyzhangx]

so your application mounted on /bitnami/postgresql/data? to change ownership, follow this guide to change uid and gid in securityContext:
https://kubernetes.io/blog/2020/12/14/kubernetes-release-1.20-fsgroupchangepolicy-fsgrouppolicy/#allow-csi-drivers-to-declare-support-for-fsgroup-based-permissions

[michae1ho11ey]

Thank you for taking the time to reply. After some experimentation I discovered the issue was the NFS server I was using had root_squash and all_squash enabled and that was what was causing the issue. Once I got the right export options set up on the NFS server things started to work swimmingly. The nfs csi plugin was working as intended.