keepalived cannot manage cluster ip natively

[elishemer]

hey

I have a multi-master bare metal configuration (HA setup)
I am using keepalived + LVS for:

1. load balancing ingress connections
2. IP fail-over

The cluster presents only one VIP address for clients to connect to.
The clients are connecting from outside of the cluster.

my kubeadm config file defines controlPlaneEndpoint: "clustervip:6443"
where clustervip resolves to that same VIP - which is maintained by keepalived

There's a chicken-and-an-egg problem with this configuration.
When I start setting up the cluster, the VIP is normally configured on one of the masters.
Everything works fine, and IP fail-over works well as well in-case the master node is taken down

However, If the entire cluster is down, and then I try to bring it back to life one node at a time - it cannot work.
And the reason is, that the kubelet cannot start, because the VIP is not configured on any of the machines, so it cannot contact the api server.
And because they cannot start, the keepalived daemonset cannot start.
So the keepalived cannot configure the ip address on the NIC.

I solved my problem outside of the keepalived project,
by creating a systemd service that makes sure to configure the ip address on one of the machines on boot, right after the network systemd scripts.

My question is whether this scenario was ever thought so? and do you have a preferred solution for this problem?
One possible solution could be to add a component to the kubelet manifests directory so it's loaded directly by it.

thanks

[steven-sheehy]

That's simply not possible to use kube-keepalived-vip as the VIP for the apiserver. This is like you said a chicken and egg problem since the apiserver depends upon kube-keepalived-vip for the vip while it depends upon the apiserver to be notified of node changes. Better to use a loadbalancer/VIP outside of kubernetes. See https://medium.com/@kvaps/creating-high-available-baremetal-kubernetes-cluster-with-kubeadm-and-keepalived-simplest-guide-71766d5e25ae

[elishemer]

The reason why i brought it up is because i think it can and should be solved internally in kubernetes. I gave two examples on how this could be achieved.

One thing i did not mention is that the api server doesnt require load balancing in this setup so we dont need to configure a port for it. The api server exposes port 6443 by default on all master nodes.
The ip failover functionally provided by keepalived is the important piece here.

One additional configuration that need to be modified is the server priority of all of the master nodes in the keepalived.conf
This way we ensure that the cluster vip will always be on one of the master nodes and the api server will be accessible through it.
Only if all of the master nodes are down, then will the ip fail over to a worker node that does not have the api server running on it. In this case, all of the other exported services in the cluster will continue to function