Better disk limit strategy

[rgaudin]

We are currently solely relying on local hostPath to store application data. So app data is stored on the k8s nodes and applications are thus tied to a specific node.

Because k8s is not meant to be used this way (it contradicts the core principle of k8s or moving pods across nodes) k8s is very aggressive when disk pressure is detected (90% of disk usage I think) and kills the running pods.

Our strategy so far is:

• have plenty of extra disk space on each node (based on expected/guessed data usage)
• have an image prune policy to purge OCI images stuff when disk usage reaches a threshold
• manually look at disk space every week in the routine.

Given we accidentally triggered DiskPressure while restoring a backup the other day (and k8s killed all pods and the ingress did not restart for a different reason), we should start discussing better strategies.

[benoit74]

Quite important indeed, probably even urgent