Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] Check for PDBs status before draining Node #49

Open
viveksyngh opened this issue Apr 2, 2021 · 4 comments
Open

[Enhancement] Check for PDBs status before draining Node #49

viveksyngh opened this issue Apr 2, 2021 · 4 comments

Comments

@viveksyngh
Copy link
Contributor

viveksyngh commented Apr 2, 2021

Is this a BUG REPORT or FEATURE REQUEST?:

enhancement

What happened:

When node reaper tries to drain a node, it does not check if all pods can be disrupted and node can be drained. It keeps trying to drain the node until command does not timesout.

What you expected to happen:

It should check before trying to drain a node.

How to reproduce it (as minimally and precisely as possible):

Try to drain a node which is having pods with 0 disruption allowed.

Anything else we need to know?:

Environment:

  • Kubernetes version:
kubectl version -o yaml

Other debugging information (if applicable):

  • relevant logs:
kubectl logs <governor-pod>
@viveksyngh viveksyngh changed the title Check for PDBs status before draining Node [Enhancement] Check for PDBs status before draining Node Apr 2, 2021
@eytan-avisror
Copy link
Collaborator

This would make a great enhancement, however would be somewhat tricky as we would have to evaluate "disruptions allowed" - since there is no way to link an existing PDB CR to a node we are about to terminate - this would mean:

  • List PDBs in the cluster
  • List Pods in the cluster
  • Map all pods to nodes that they are scheduled on
  • Before terminating a node, evaluate whether a disruption is likely allowed (e.g. all pods are running / enough pods are running)

We can probably add a flag to enable/disable this feature

@shrinandj
Copy link
Collaborator

Another option might be to do kubectl drain --dry-run and see if that call succeeds. If there are PDBs that might get violated, the dry-run would fail (need to test this though).

@eytan-avisror
Copy link
Collaborator

Another option might be to do kubectl drain --dry-run and see if that call succeeds. If there are PDBs that might get violated, the dry-run would fail (need to test this though).

Good idea worth a check

@viveksyngh
Copy link
Contributor Author

viveksyngh commented Apr 5, 2021

kubectl drain node --dry-run=client

does not check if it can disrupt the pod or not. It just print evicting for all pods on that node regardless of their pdb status.

I tried this on one of the node which node-reaper was failed to drain and has some pods which can be disrupted.

@eytan-avisror @shrinandj

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants