How to contact a VIP kube-vip in a HA cluster under VPN wireguard

[ByAziX]

Cluster information:

Kubernetes version: k3s version 1.19.1
Cloud being used: (put bare-metal if not on a public cloud) : On contabo VPS
Installation method: https://github.com/techno-tim/k3s-ansible
Host OS: Ubuntu version 22.04
CNI and version: flannel

Hello,

I have 5 VPS I wanted to have some fun and make a kubernetes HA cluster. I am a beginner in the world of kubernetes.
So my nodes communicate through a vpn tunnel under wireguard.

I have 3 masters 10.0.0.1 and 10.0.0.2 and 10.0.0.3
and
2 workers in 10.0.0.3 and 10.0.0.5.
and a VIP (kube-vip) in 10.0.0.200

all my flows go through the wg0 interface for traffic in 10.0.0.0/24

My wireguard github: https://github.com/ByAziX/ansible-wireguard

The problem is that my worker and master nodes manage to communicate with each other via the VPN, but when I decide to have my workers communicate with the VIP, there's no response from the VIP.

I think I'm misconfiguring the Kube-VIP in my cluster.
I'm also wondering about using BGP to have dynamic routes depending on the nodes and for HA.
If someone can explain me the BGP with Kube-vip or how can i solve the problem please

ping:

The ansible remains blocked when k3s-nodes service is trying to start and has to fetch a curl of the 10.0.0.200 cert:

k3s-node service

~# kubectl get pods -n kube-system

NAME READY STATUS RESTARTS AGE
coredns-576bfc4dc7-8hdm2 1/1 Running 1 (8m48s ago) 10m
kube-vip-ds-bppf4 1/1 Running 1 (8m41s ago) 9m28s
kube-vip-ds-r7r8t 1/1 Running 1 (8m48s ago) 10m
kube-vip-ds-wnmp8 1/1 Running 1 (8m49s ago) 9m55s
local-path-provisioner-86f46b7bf7-gw2zr 1/1 Running 1 (8m48s ago) 10m
metrics-server-557ff575fb-tn9d5 1/1 Running 1 (8m48s ago) 10m

master 1:

kubectl get daemonset kube-vip-ds -n kube-system -o yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
annotations:
deprecated.daemonset.template.generation: "1"
objectset.rio.cattle.io/applied: H4sIAAAAAAAA/5yV32/jNgzH/5WBz27OTn/dGdhDsPah2BocLsWGoSgKWqIdLbLkUbTboPD/PshJWydp7g5DXiTx4y8pimReABvzJ3Ew3kEO2DThU5dBAivjNORwhVR7tyCBBGoS1CgI+Qugc15QjHchbn3xDykJJBM2fqJQxNLE+E8makBy1O6fHPFJ1a0gh9VpGFm6LPnld+P0rzOtvfuhhMOaIIfOND+FhgZV5FdtQSdhHYRq6BOwWJD97oWWGJaQQzbVafrlMk31l7IoLstUKXVK5UWBGabnl+nnL/oyzcqLKLoNbXDVmeZEB9gcHgkiNKRiCIEsKfEc1zWKWv7xFt0Hkn2fgFDdWBQavhg9lv3Rh68usSyNM7IeUK9pNtoz/dsaJn3VsnHVQi1Jt9a46qZy/u34+plUK0MtbRQW20vcEdcB8vvtVa6fG6YQNuVz/wIrWkM+fHDC3tIkBsiOhEJMeo1BiOPDNsQ4pASun02QAP1Dn/wvTeWdsLcnjUVHx6Qf+pidiKJxxBtd5CouoEaHFTE8JECuG0zvVfiIHCuxQ9vGkxJtIIihbpGiah7JYWFpRAm3O1DjWUbmi7Oz07E5ujFOiMtYR+/cU5XuY8poHhGn0zGgfiIU1Ty+l+w7tlu6Oy61duF7GQidOnR7QEUlS6iJh0Iywyg4FuQWDqRbxj02O98nmRw9aUJtjRvHkB1kj0l43RAbr0fcTg5R61h8OzKT+Jum6f67s2+F2Iy1ssvpQGf7bEOx7A5Uz/KLs/Nsmmy355st9A8JmBqrSFZLxbHQX3v9bZF36eTzZApb9Gtr7VdvjYrdMrNPuH6dT6NBAQkwBd+yojhH4sgg1bKR9W/eCT1L7HeFDRbGGjEDFHMS+2R+ffc4u7q9mUMyrL/N/oIEFn8vHu9ubq8h9thDAksfZE7y5HkFeXzY6II7o2imlG+dzA8iEm+JX/+D7l+AypKUQA5zvx1PRxo72WE3M+voDOgTaBuNQgthFKriMBxmpqC0wz1Vy0xO5m1dEL961pCnCWgKcWZ+ZHLD2a0J4YPjb4R6DXna9/8FAAD//z936YWhBwAA
objectset.rio.cattle.io/id: ""
objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
objectset.rio.cattle.io/owner-name: vip
objectset.rio.cattle.io/owner-namespace: kube-system
creationTimestamp: "2024-12-11T18:45:49Z"
generation: 1
labels:
objectset.rio.cattle.io/hash: 12d009700d9fbb7f0ccc3ef6ba1a057089d701f6
name: kube-vip-ds
namespace: kube-system
resourceVersion: "1326"
uid: d547d958-84bb-47e4-8333-efbd68f614bf
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
name: kube-vip-ds
template:
metadata:
creationTimestamp: null
labels:
name: kube-vip-ds
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/master
operator: Exists
- matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: Exists
containers:
- args:
- manager
env:
- name: vip_arp
value: "false"
- name: bgp_enable
value: "true"
- name: port
value: "6443"
- name: vip_interface
value: wg0
- name: vip_cidr
value: "32"
- name: cp_enable
value: "true"
- name: cp_namespace
value: kube-system
- name: vip_ddns
value: "false"
- name: svc_enable
value: "false"
- name: vip_leaderelection
value: "true"
- name: vip_leaseduration
value: "15"
- name: vip_renewdeadline
value: "10"
- name: vip_retryperiod
value: "2"
- name: address
value: 10.0.0.200
- name: bgp_routerid
value: 10.0.0.1
- name: bgp_peers
value: 10.0.0.4:64512,10.0.0.5:64512
image: ghcr.io/kube-vip/kube-vip:v0.8.2
imagePullPolicy: Always
name: kube-vip
resources: {}
securityContext:
capabilities:
add:
- NET_ADMIN
- NET_RAW
- SYS_TIME
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
hostNetwork: true
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: kube-vip
serviceAccountName: kube-vip
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoSchedule
operator: Exists
- effect: NoExecute
operator: Exists
updateStrategy:
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
type: RollingUpdate
status:
currentNumberScheduled: 3
desiredNumberScheduled: 3
numberAvailable: 3
numberMisscheduled: 0
numberReady: 3
observedGeneration: 1
updatedNumberScheduled: 3

master 2:
root@vmi2333090:~# kubectl get daemonset kube-vip-ds -n kube-system -o yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
annotations:
deprecated.daemonset.template.generation: "1"
objectset.rio.cattle.io/applied: H4sIAAAAAAAA/5yV32/jNgzH/5WBz27OTn/dGdhDsPah2BocLsWGoSgKWqIdLbLkUbTboPD/PshJWydp7g5DXiTx4y8pimReABvzJ3Ew3kEO2DThU5dBAivjNORwhVR7tyCBBGoS1CgI+Qugc15QjHchbn3xDykJJBM2fqJQxNLE+E8makBy1O6fHPFJ1a0gh9VpGFm6LPnld+P0rzOtvfuhhMOaIIfOND+FhgZV5FdtQSdhHYRq6BOwWJD97oWWGJaQQzbVafrlMk31l7IoLstUKXVK5UWBGabnl+nnL/oyzcqLKLoNbXDVmeZEB9gcHgkiNKRiCIEsKfEc1zWKWv7xFt0Hkn2fgFDdWBQavhg9lv3Rh68usSyNM7IeUK9pNtoz/dsaJn3VsnHVQi1Jt9a46qZy/u34+plUK0MtbRQW20vcEdcB8vvtVa6fG6YQNuVz/wIrWkM+fHDC3tIkBsiOhEJMeo1BiOPDNsQ4pASun02QAP1Dn/wvTeWdsLcnjUVHx6Qf+pidiKJxxBtd5CouoEaHFTE8JECuG0zvVfiIHCuxQ9vGkxJtIIihbpGiah7JYWFpRAm3O1DjWUbmi7Oz07E5ujFOiMtYR+/cU5XuY8poHhGn0zGgfiIU1Ty+l+w7tlu6Oy61duF7GQidOnR7QEUlS6iJh0Iywyg4FuQWDqRbxj02O98nmRw9aUJtjRvHkB1kj0l43RAbr0fcTg5R61h8OzKT+Jum6f67s2+F2Iy1ssvpQGf7bEOx7A5Uz/KLs/Nsmmy355st9A8JmBqrSFZLxbHQX3v9bZF36eTzZApb9Gtr7VdvjYrdMrNPuH6dT6NBAQkwBd+yojhH4sgg1bKR9W/eCT1L7HeFDRbGGjEDFHMS+2R+ffc4u7q9mUMyrL/N/oIEFn8vHu9ubq8h9thDAksfZE7y5HkFeXzY6II7o2imlG+dzA8iEm+JX/+D7l+AypKUQA5zvx1PRxo72WE3M+voDOgTaBuNQgthFKriMBxmpqC0wz1Vy0xO5m1dEL961pCnCWgKcWZ+ZHLD2a0J4YPjb4R6DXna9/8FAAD//z936YWhBwAA
objectset.rio.cattle.io/id: ""
objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
objectset.rio.cattle.io/owner-name: vip
objectset.rio.cattle.io/owner-namespace: kube-system
creationTimestamp: "2024-12-11T18:45:49Z"
generation: 1
labels:
objectset.rio.cattle.io/hash: 12d009700d9fbb7f0ccc3ef6ba1a057089d701f6
name: kube-vip-ds
namespace: kube-system
resourceVersion: "1326"
uid: d547d958-84bb-47e4-8333-efbd68f614bf
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
name: kube-vip-ds
template:
metadata:
creationTimestamp: null
labels:
name: kube-vip-ds
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/master
operator: Exists
- matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: Exists
containers:
- args:
- manager
env:
- name: vip_arp
value: "false"
- name: bgp_enable
value: "true"
- name: port
value: "6443"
- name: vip_interface
value: wg0
- name: vip_cidr
value: "32"
- name: cp_enable
value: "true"
- name: cp_namespace
value: kube-system
- name: vip_ddns
value: "false"
- name: svc_enable
value: "false"
- name: vip_leaderelection
value: "true"
- name: vip_leaseduration
value: "15"
- name: vip_renewdeadline
value: "10"
- name: vip_retryperiod
value: "2"
- name: address
value: 10.0.0.200
- name: bgp_routerid
value: 10.0.0.1
- name: bgp_peers
value: 10.0.0.4:64512,10.0.0.5:64512
image: ghcr.io/kube-vip/kube-vip:v0.8.2
imagePullPolicy: Always
name: kube-vip
resources: {}
securityContext:
capabilities:
add:
- NET_ADMIN
- NET_RAW
- SYS_TIME
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
hostNetwork: true
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: kube-vip
serviceAccountName: kube-vip
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoSchedule
operator: Exists
- effect: NoExecute
operator: Exists
updateStrategy:
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
type: RollingUpdate
status:
currentNumberScheduled: 3
desiredNumberScheduled: 3
numberAvailable: 3
numberMisscheduled: 0
numberReady: 3
observedGeneration: 1
updatedNumberScheduled: 3

Variables Used

all.yml

---
k3s_version: v1.30.2+k3s2
ansible_user: root
systemd_dir: /etc/systemd/system

# Configuration générale
system_timezone: UTC # Remplacez par votre timezone

# Interface pour flannel
flannel_iface: eth0

# Cluster CIDR pour les pods
cluster_cidr: 10.52.0.0/16

# Configuration Calico (désactivée par défaut)
# calico_iface: "eth0"
calico_ebpf: false
calico_tag: v3.28.0

# Configuration Cilium (désactivée par défaut)
# cilium_iface: "eth0"
cilium_mode: native
cilium_tag: v1.16.0
cilium_hubble: true
cilium_bgp: false
cilium_bgp_my_asn: "64513"
cilium_bgp_peer_asn: "64512"
cilium_bgp_peer_address: 10.0.0.100
cilium_bgp_lb_cidr: 10.0.0.0/24

# Configuration kube-vip
kube_vip_arp: false  # ARP désactivé car vous utilisez BGP
kube_vip_iface: wg0  # Interface réseau utilisée pour la VIP (WireGuard)
kube_vip_bgp: true  # Active le mode BGP
kube_vip_bgp_routerid: "10.0.0.1"
kube_vip_bgp_as: "64513"  # AS local pour BGP
kube_vip_bgp_peers:  # Liste des pairs BGP
  - peer_address: 10.0.0.4  # Adresse du worker 1
    peer_asn: "64512"  # AS du worker
  - peer_address: 10.0.0.5  # Adresse du worker 2
    peer_asn: "64512"  # AS du worker
apiserver_endpoint: 10.0.0.200  # VIP pour l'API server
k3s_token: some-SUPER-DEDEUPER-secret-password


# Configuration des nœuds
k3s_node_ip: "{{ ansible_facts[(cilium_iface | default(calico_iface | default(flannel_iface)))]['ipv4']['address'] }}"
k3s_master_taint: "{{ true if groups['node'] | default([]) | length >= 1 else false }}"
extra_args: >-
  {{ '--flannel-iface=' + flannel_iface if calico_iface is not defined and cilium_iface is not defined else '' }}
  --node-ip={{ k3s_node_ip }}

# Arguments supplémentaires pour le serveur et les agents
extra_server_args: >-
  {{ extra_args }}
  {{ '--node-taint node-role.kubernetes.io/master=true:NoSchedule' if k3s_master_taint else '' }}
  {% if calico_iface is defined or cilium_iface is defined %}
  --flannel-backend=none
  --disable-network-policy
  --cluster-cidr={{ cluster_cidr | default('10.52.0.0/16') }}
  {% endif %}
  --tls-san {{ apiserver_endpoint }}
  --disable servicelb
  --disable traefik
extra_agent_args: >-
  {{ extra_args }}

# Image de kube-vip
kube_vip_tag_version: v0.8.2

# Configuration MetalLB
metal_lb_type: native
metal_lb_mode: layer2
metal_lb_ip_range: 10.0.0.80-10.0.0.90
metal_lb_speaker_tag_version: v0.14.8
metal_lb_controller_tag_version: v0.14.8

# Configuration Proxmox LXC (désactivée par défaut)
proxmox_lxc_configure: false
proxmox_lxc_ssh_user: root
proxmox_lxc_ct_ids:
  - 200
  - 201
  - 202
  - 203
  - 204

# Registry personnalisé (désactivé par défaut)
custom_registries: false
custom_registries_yaml: |
  mirrors:
    docker.io:
      endpoint:
        - "https://registry.domain.com/v2/dockerhub"
    quay.io:
      endpoint:
        - "https://registry.domain.com/v2/quayio"
    ghcr.io:
      endpoint:
        - "https://registry.domain.com/v2/ghcrio"
    registry.domain.com:
      endpoint:
        - "https://registry.domain.com"

  configs:
    "registry.domain.com":
      auth:
        username: yourusername
        password: yourpassword

# Commande de redémarrage personnalisée (désactivée par défaut)
# custom_reboot_command: /usr/sbin/shutdown -r now

# Configuration du proxy (désactivée par défaut)
# proxy_env:
#   HTTP_PROXY: "http://proxy.domain.local:3128"
#   HTTPS_PROXY: "http://proxy.domain.local:3128"
#   NO_PROXY: "*.domain.local,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"

Hosts

host.ini

[master]
10.0.0.1
10.0.0.2
10.0.0.3

[node]
10.0.0.4
10.0.0.5

[k3s_cluster:children]
master
node