Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more ways to track kubernetes nodes rotation #2823

Open
baznikin opened this issue Dec 20, 2024 · 0 comments
Open

Add more ways to track kubernetes nodes rotation #2823

baznikin opened this issue Dec 20, 2024 · 0 comments

Comments

@baznikin
Copy link

Operator have very nice feature node_readiness_label docs to track if node where master runs under upgrade procedure. However current implementation work only for some cloud providers. I already mentioned it here
If operator move master to other replica more accurately and smoothly than just allowing Kubernetes to force drain a node (when provider eventually give up on PDB and drain the node anyway) or removing master replica PDB and push your luck (k8s can kill master first, then kill new master etc - you can't predict rotation order), I believe it could and should be addressed.

Citing myself from previous issue:

I wanted to implement node_readiness_label on my clusters, but during testing I found out my managed k8s provider (DigitalOcean) doesn't set any labels for Ready nodes. Instead it set annotation for nodes, which are drained:

metadata:
  annotations:
    k8s.digitalocean.com/drain-start-time: "2023-01-13T12:22:07Z"

So, there is few possible use cases:

  1. label added for Ready node, removed when node is going to recycle (addressed by node_readiness_label)
  2. label removed for Ready node, added when node is going to recycle
  3. annotation added for Ready node, removed when node is going to recycle
  4. annotation removed for Ready node, added when node is going to recycle (my case)

Only 1st case covered by node_readiness_label feature. It should be addressed somehow.

Simplest solution is watch for node spec.unschedulable==true or when taint effect: NoSchedule; key: node.kubernetes.io/unschedulable appeared. It have drawback - is engineer kubectl cordon node for some purpose master will be rotated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant