snapshots

[mscrocdile]

Hi, i've installed nfs https://microk8s.io/docs/how-to-nfs
It works.
Then i installed k10 and wanted to make snapshot. However it refused it.
I've enabled externalsnapshotter in default values of csi-driver-nfs chart and created VolumeSnapshotClass.
Is there anything else i should do? Do i have to install that external snapshooter from here https://github.com/kubernetes-csi/external-snapshotter? If so, is there any tutorial for that? I cannot see the wood through the trees.

k10 error:
Failed to create snapshot: failed to take snapshot of the volume 10.0.0.6#nfsserver/pv#pvc-c5cec9f6-1555-442c-bd66-c984cad71d16##: \"rpc error: code = Internal desc = failed to create archive for snapshot: exit status 1: ./\\n./pg_replslot/\\n./pg_logical/\\n./pg_logical/replorigin_checkpoint\\n./pg_logical/mappings/\\n./pg_logical/snapshots/\\n./pg_snaps ...

[andyzhangx]

what's the full logs of csi driver controller? you could get logs by https://github.com/kubernetes-csi/csi-driver-nfs/blob/master/docs/csi-debug.md#case1-volume-createdelete-failed

[mscrocdile]

There is nothing suspicious in logs. In case i create snapshot of another pvc(same nfs storage class) it works. Problem is only with PVC where postgres database of zabbix is.

[wozniakjan]

in that case, I would suspect timeout. The snapshots are implemented as compressed tarballs and for bigger volumes, this frequently takes longer than default CSI timeouts

Do you see any context deadline exceeded in the logs?

For more context on how to debug, you can take a look at: #509
This should help you bump the default timeout: #512

[mscrocdile]

snapshot_controller.log
snapshot_describe.log
There is no timeout error. I've added complete snapshot description and snapshot controller log (with some error and also proof that snapshot of another pvc is ready).

[sps48247]

Did you find a solution to this?

[andyzhangx]

current timeout is 1200s, maybe you could increase the timeout as 2400 to try again? If that works, I could increase the default value.

https://github.com/kubernetes-csi/csi-driver-nfs/blob/9bcde34e7d0492878a4136162a5fb6f0944c9d36/charts/latest/csi-driver-nfs/templates/csi-nfs-controller.yaml#L88C18-L88C31

[Azam-bhatti]

I am getting below error during creating snapshot. Is NFS CSI driver take snapshot of in use PVCs?
rpc
error: code = Internal desc = failed to create archive for snapshot: exit
status 1:
./\n./logs/\n./logs/.lock\n./logs/recovery-point-offset-checkpoint\n./logs/log-start-offset-checkpoint\n./logs/cleaner-offset-checkpoint\n./logs/replication-offset-checkpoint\n./logs/meta.properties\n./logs/__confluent.support.metrics-0/\n./logs/__confluent.support.metrics-0/leader-epoch-checkpoint\n./logs/__confluent.support.metrics-0/00000000000000000000.log\n./logs/__confluent.support.metrics-0/00000000000000000000.index\n./logs/__confluent.support.metrics-0/00000000000000000000.timeindex\n./logs/__confluent.support.metrics-0/00000000000000000002.snapshot\n./logs/_schemas-0/\n./logs/_schemas-0/leader-epoch-checkpoint\n./logs/_schemas-0/00000000000000000000.log\n./logs/_schemas-0/00000000000000000000.index\n./logs/_schemas-0/00000000000000000000.timeindex\n./logs/_schemas-0/00000000000000000004.snapshot\n./logs/__consumer_offsets-0/\n./logs/__consumer_offsets-0/leader-epoch-checkpoint\n./logs/__consumer_offsets-0/00000000000000000000.log\n./logs/__consumer_offsets-0/00000000000000000000.index\n./logs/__consumer_offsets-0/00000000000000000000.timeindex\n./logs/__consumer_offsets-48/\n./logs/__consumer_offsets-48/leader-epoch-checkpoint\n./logs/__consumer_offsets-48/00000000000000000000.log\n./logs/__consumer_offsets-48/00000000000000000000.index\n./logs/__consumer_offsets-48/00000000000000000000.timeindex\n./logs/__consumer_offsets-45/\n./logs/__consumer_offsets-45/leader-epoch-checkpoint\n./logs/__consumer_offsets-45/00000000000000000000.log\n./logs/__consumer_offsets-45/00000000000000000000.index\n./logs/__consumer_offsets-45/00000000000000000000.timeindex\n./logs/__consumer_offsets-42/\n./logs/__consumer_offsets-42/leader-epoch-checkpoint\n./logs/__consumer_offsets-42/00000000000000000000.log\n./logs/__consumer_offsets-42/00000000000000000000.index\n./logs/__consumer_offsets-42/00000000000000000000.timeindex\n./logs/__consumer_offsets-39/\n./logs/__consumer_offsets-39/leader-epoch-checkpoint\n./logs/__consumer_offsets-39/00000000000000000000.log\n./logs/__consumer_offsets-39/00000000000000000000.index\n./logs/__consumer_offsets-39/00000000000000000000.timeindex\n./logs/__consumer_offsets-36/\n./logs/__consumer_offsets-36/leader-epoch-checkpoint\n./logs/__consumer_offsets-36/00000000000000000000.log\n./logs/__consumer_offsets-36/00000000000000000000.index\n./logs/__consumer_offsets-36/00000000000000000000.timeindex\n./logs/__consumer_offsets-33/\n./logs/__consumer_offsets-33/leader-epoch-checkpoint\n./logs/__consumer_offsets-33/00000000000000000000.log\n./logs/__consumer_offsets-33/00000000000000000000.index\n./logs/__consumer_offsets-33/00000000000000000000.timeindex\n./logs/__consumer_offsets-30/\n./logs/__consumer_offsets-30/leader-epoch-checkpoint\n./logs/__consumer_offsets-30/00000000000000000000.log\n./logs/__consumer_offsets-30/00000000000000000000.index\n./logs/__consumer_offsets-30/00000000000000000000.timeindex\n./logs/__consumer_offsets-27/\n./logs/__consumer_offsets-27/leader-epoch-checkpoint\n./logs/__consumer_offsets-27/00000000000000000000.log\n./logs/__consumer_offsets-27/00000000000000000000.index\n./logs/__consumer_offsets-27/00000000000000000000.timeindex\n./logs/__consumer_offsets-24/\n./logs/__consumer_offsets-24/leader-epoch-checkpoint\n./logs/__consumer_offsets-24/00000000000000000000.log\n./logs/__consumer_offsets-24/00000000000000000000.index\n./logs/__consumer_offsets-24/00000000000000000000.timeindex\n./logs/__consumer_offsets-21/\n./logs/__consumer_offsets-21/leader-epoch-checkpoint\n./logs/__consumer_offsets-21/00000000000000000000.log\n./logs/__consumer_offsets-21/00000000000000000000.index\n./logs/__consumer_offsets-21/00000000000000000000.timeindex\n./logs/__consumer_offsets-18/\n./logs/__consumer_offsets-18/leader-epoch-checkpoint\n./logs/__consumer_offsets-18/00000000000000000000.log\n./logs/__consumer_offsets-18/00000000000000000000.index\n./logs/__consumer_offsets-18/00000000000000000000.timeindex\n./logs/__consumer_offsets-15/\n./logs/__consumer_offsets-15/leader-epoch-checkpoint\n./logs/__consumer_offsets-15/00000000000000000000.log\n./logs/__consumer_offsets-15/00000000000000000000.index\n./logs/__consumer_offsets-15/00000000000000000000.timeindex\n./logs/__consumer_offsets-12/\n./logs/__consumer_offsets-12/leader-epoch-checkpoint\n./logs/__consumer_offsets-12/00000000000000000000.log\n./logs/__consumer_offsets-12/00000000000000000000.index\n./logs/__consumer_offsets-12/00000000000000000000.timeindex\n./logs/__consumer_offsets-9/\n./logs/__consumer_offsets-9/leader-epoch-checkpoint\n./logs/__consumer_offsets-9/00000000000000000000.log\n./logs/__consumer_offsets-9/00000000000000000000.index\n./logs/__consumer_offsets-9/00000000000000000000.timeindex\n./logs/__consumer_offsets-6/\n./logs/__consumer_offsets-6/leader-epoch-checkpoint\n./logs/__consumer_offsets-6/00000000000000000000.log\n./logs/__consumer_offsets-6/00000000000000000000.index\n./logs/__consumer_offsets-6/00000000000000000000.timeindex\n./logs/__consumer_offsets-3/\n./logs/__consumer_offsets-3/leader-epoch-checkpoint\n./logs/__consumer_offsets-3/00000000000000000000.log\n./logs/__consumer_offsets-3/00000000000000000000.index\n./logs/__consumer_offsets-3/00000000000000000000.timeindex\n./logs/__consumer_offsets-29/\n./logs/__consumer_offsets-29/leader-epoch-checkpoint\n./logs/__consumer_offsets-29/00000000000000000000.log\n./logs/__consumer_offsets-29/00000000000000000000.index\n./logs/__consumer_offsets-29/00000000000000000000.timeindex\n./logs/__consumer_offsets-29/00000000000000000003.snapshot\n./logs/__consumer_offsets-10/\n./logs/__consumer_offsets-10/leader-epoch-checkpoint\n./logs/__consumer_offsets-10/00000000000000000000.log\n./logs/__consumer_offsets-10/00000000000000000000.index\n./logs/__consumer_offsets-10/00000000000000000000.timeindex\n./logs/__consumer_offsets-26/\n./logs/__consumer_offsets-26/leader-epoch-checkpoint\n./logs/__consumer_offsets-26/00000000000000000000.log\n./logs/__consumer_offsets-26/00000000000000000000.index\n./logs/__consumer_offsets-26/00000000000000000000.timeindex\n./logs/__consumer_offsets-7/\n./logs/__consumer_offsets-7/leader-epoch-checkpoint\n./logs/__consumer_offsets-7/00000000000000000000.log\n./logs/__consumer_offsets-7/00000000000000000000.index\n./logs/__consumer_offsets-7/00000000000000000000.timeindex\n./logs/__consumer_offsets-4/\n./logs/__consumer_offsets-4/leader-epoch-checkpoint\n./logs/__consumer_offsets-4/00000000000000000000.log\n./logs/__consumer_offsets-4/00000000000000000000.index\n./logs/__consumer_offsets-4/00000000000000000000.timeindex\n./logs/__consumer_offsets-23/\n./logs/__consumer_offsets-23/leader-epoch-checkpoint\n./logs/__consumer_offsets-23/00000000000000000000.log\n./logs/__consumer_offsets-23/00000000000000000000.index\n./logs/__consumer_offsets-23/00000000000000000000.timeindex\n./logs/__consumer_offsets-1/\n./logs/__consumer_offsets-1/leader-epoch-checkpoint\n./logs/__consumer_offsets-1/00000000000000000000.log\n./logs/__consumer_offsets-1/00000000000000000000.index\n./logs/__consumer_offsets-1/00000000000000000000.timeindex\n./logs/__consumer_offsets-20/\n./logs/__consumer_offsets-20/leader-epoch-checkpoint\n./logs/__consumer_offsets-20/00000000000000000000.log\n./logs/__consumer_offsets-20/00000000000000000000.index\n./logs/__consumer_offsets-20/00000000000000000000.timeindex\n./logs/__consumer_offsets-17/\n./logs/__consumer_offsets-17/leader-epoch-checkpoint\n./logs/__consumer_offsets-17/00000000000000000000.log\n./logs/__consumer_offsets-17/00000000000000000000.index\n./logs/__consumer_offsets-17/00000000000000000000.timeindex\n./logs/__consumer_offsets-14/\n./logs/__consumer_offsets-14/leader-epoch-checkpoint\n./logs/__consumer_offsets-14/00000000000000000000.log\n./logs/__consumer_offsets-14/00000000000000000000.index\n./logs/__consumer_offsets-14/00000000000000000000.timeindex\n./logs/__consumer_offsets-49/\n./logs/__consumer_offsets-49/leader-epoch-checkpoint\n./logs/__consumer_offsets-49/00000000000000000000.log\n./logs/__consumer_offsets-49/00000000000000000000.index\n./logs/__consumer_offsets-49/00000000000000000000.timeindex\n./logs/__consumer_offsets-11/\n./logs/__consumer_offsets-11/leader-epoch-checkpoint\n./logs/__consumer_offsets-11/00000000000000000000.log\n./logs/__consumer_offsets-11/00000000000000000000.index\n./logs/__consumer_offsets-11/00000000000000000000.timeindex\n./logs/__consumer_offsets-46/\n./logs/__consumer_offsets-46/leader-epoch-checkpoint\n./logs/__consumer_offsets-46/00000000000000000000.log\n./logs/__consumer_offsets-46/00000000000000000000.index\n./logs/__consumer_offsets-46/00000000000000000000.timeindex\n./logs/__consumer_offsets-8/\n./logs/__consumer_offsets-8/leader-epoch-checkpoint\n./logs/__consumer_offsets-8/00000000000000000000.log\n./logs/__consumer_offsets-8/00000000000000000000.index\n./logs/__consumer_offsets-8/00000000000000000000.timeindex\n./logs/__consumer_offsets-43/\n./logs/__consumer_offsets-43/leader-epoch-checkpoint\n./logs/__consumer_offsets-43/00000000000000000000.log\n./logs/__consumer_offsets-43/00000000000000000000.index\n./logs/__consumer_offsets-43/00000000000000000000.timeindex\n./logs/__consumer_offsets-5/\n./logs/__consumer_offsets-5/leader-epoch-checkpoint\n./logs/__consumer_offsets-5/00000000000000000000.log\n./logs/__consumer_offsets-5/00000000000000000000.index\n./logs/__consumer_offsets-5/00000000000000000000.timeindex\n./logs/__consumer_offsets-40/\n./logs/__consumer_offsets-40/leader-epoch-checkpoint\n./logs/__consumer_offsets-40/00000000000000000000.log\n./logs/__consumer_offsets-40/00000000000000000000.index\n./logs/__consumer_offsets-40/00000000000000000000.timeindex\n./logs/__consumer_offsets-2/\n./logs/__consumer_offsets-2/leader-epoch-checkpoint\n./logs/__consumer_offsets-2/00000000000000000000.log\n./logs/__consumer_offsets-2/00000000000000000000.index\n./logs/__consumer_offsets-2/00000000000000000000.timeindex\n./logs/__consumer_offsets-37/\n./logs/__consumer_offsets-37/leader-epoch-checkpoint\n./logs/__consumer_offsets-37/00000000000000000000.log\n./logs/__consumer_offsets-37/00000000000000000000.index\n./logs/__consumer_offsets-37/00000000000000000000.timeindex\n./logs/__consumer_offsets-34/\n./logs/__consumer_offsets-34/leader-epoch-checkpoint\n./logs/__consumer_offsets-34/00000000000000000000.log\n./logs/__consumer_offsets-34/00000000000000000000.index\n./logs/__consumer_offsets-34/00000000000000000000.timeindex\n./logs/__consumer_offsets-31/\n./logs/__consumer_offsets-31/leader-epoch-checkpoint\n./logs/__consumer_offsets-31/00000000000000000000.log\n./logs/__consumer_offsets-31/00000000000000000000.index\n./logs/__consumer_offsets-31/00000000000000000000.timeindex\n./logs/__consumer_offsets-47/\n./logs/__consumer_offsets-47/leader-epoch-checkpoint\n./logs/__consumer_offsets-47/00000000000000000000.log\n./logs/__consumer_offsets-47/00000000000000000000.index\n./logs/__consumer_offsets-47/00000000000000000000.timeindex\n./logs/__consumer_offsets-19/\n./logs/__consumer_offsets-19/leader-epoch-checkpoint\n./logs/__consumer_offsets-19/00000000000000000000.log\n./logs/__consumer_offsets-19/00000000000000000000.index\n./logs/__consumer_offsets-19/00000000000000000000.timeindex\n./logs/__consumer_offsets-28/\n./logs/__consumer_offsets-28/leader-epoch-checkpoint\n./logs/__consumer_offsets-28/00000000000000000000.log\n./logs/__consumer_offsets-28/00000000000000000000.index\n./logs/__consumer_offsets-28/00000000000000000000.timeindex\n./logs/__consumer_offsets-38/\n./logs/__consumer_offsets-38/leader-epoch-checkpoint\n./logs/__consumer_offsets-38/00000000000000000000.log\n./logs/__consumer_offsets-38/00000000000000000000.index\n./logs/__consumer_offsets-38/00000000000000000000.timeindex\n./logs/__consumer_offsets-35/\n./logs/__consumer_offsets-35/leader-epoch-checkpoint\n./logs/__consumer_offsets-35/00000000000000000000.log\n./logs/__consumer_offsets-35/00000000000000000000.index\n./logs/__consumer_offsets-35/00000000000000000000.timeindex\n./logs/__consumer_offsets-44/\n./logs/__consumer_offsets-44/leader-epoch-checkpoint\n./logs/__consumer_offsets-44/00000000000000000000.log\n./logs/__consumer_offsets-44/00000000000000000000.index\n./logs/__consumer_offsets-44/00000000000000000000.timeindex\n./logs/__consumer_offsets-25/\n./logs/__consumer_offsets-25/leader-epoch-checkpoint\n./logs/__consumer_offsets-25/00000000000000000000.log\n./logs/__consumer_offsets-25/00000000000000000000.index\n./logs/__consumer_offsets-25/00000000000000000000.timeindex\n./logs/__consumer_offsets-16/\n./logs/__consumer_offsets-16/leader-epoch-checkpoint\n./logs/__consumer_offsets-16/00000000000000000000.log\n./logs/__consumer_offsets-16/00000000000000000000.index\n./logs/__consumer_offsets-16/00000000000000000000.timeindex\n./logs/__consumer_offsets-22/\n./logs/__consumer_offsets-22/leader-epoch-checkpoint\n./logs/__consumer_offsets-22/00000000000000000000.log\n./logs/__consumer_offsets-22/00000000000000000000.index\n./logs/__consumer_offsets-22/00000000000000000000.timeindex\n./logs/__consumer_offsets-41/\n./logs/__consumer_offsets-41/leader-epoch-checkpoint\n./logs/__consumer_offsets-41/00000000000000000000.log\n./logs/__consumer_offsets-41/00000000000000000000.index\n./logs/__consumer_offsets-41/00000000000000000000.timeindex\n./logs/__consumer_offsets-32/\n./logs/__consumer_offsets-32/leader-epoch-checkpoint\n./logs/__consumer_offsets-32/00000000000000000000.log\n./logs/__consumer_offsets-32/00000000000000000000.index\n./logs/__consumer_offsets-32/00000000000000000000.timeindex\n./logs/__consumer_offsets-13/\n./logs/__consumer_offsets-13/leader-epoch-checkpoint\n./logs/__consumer_offsets-13/00000000000000000000.log\n./logs/__consumer_offsets-13/00000000000000000000.index\n./logs/__consumer_offsets-13/00000000000000000000.timeindex\ntar:
./logs: file changed as we read it\n"

[andyzhangx]

this nfs csi driver is using tar command to create snapshot, so when taking snapshot, the file content should be not in writing state.