10.43.0.1 inaccessible from workers pods

[DamianoSamperi]

Environmental Info:
K3s Version:

kubectl get nodes -o wide
NAME                  STATUS   ROLES                  AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
k3s-controlplane-01   Ready    control-plane,master   15d   v1.31.4+k3s1   192.168.1.150   <none>        Ubuntu 24.04.1 LTS   6.8.0-45-generic   containerd://1.7.23-k3s2
nano90                Ready    <none>                 15d   v1.31.4+k3s1   192.168.1.90    <none>        Ubuntu 18.04.6 LTS   4.9.337-tegra      containerd://1.7.23-k3s2
nano91                Ready    <none>                 15d   v1.31.4+k3s1   192.168.1.91    <none>        Ubuntu 18.04.6 LTS   4.9.253-tegra      containerd://1.7.23-k3s2
nano92                Ready    <none>                 15d   v1.31.4+k3s1   192.168.1.92    <none>        Ubuntu 18.04.6 LTS   4.9.253-tegra      containerd://1.7.23-k3s2
nano93                Ready    <none>                 15d   v1.31.4+k3s1   192.168.1.93    <none>        Ubuntu 18.04.6 LTS   4.9.253-tegra      containerd://1.7.23-k3s2
nano94                Ready    <none>                 15d   v1.31.4+k3s1   192.168.1.94    <none>        Ubuntu 18.04.6 LTS   4.9.253-tegra      containerd://1.7.23-k3s2
nano95                Ready    <none>                 15d   v1.31.4+k3s1   192.168.1.95    <none>        Ubuntu 18.04.6 LTS   4.9.253-tegra      containerd://1.7.23-k3s2
nano96                Ready    <none>                 15d   v1.31.4+k3s1   192.168.1.96    <none>        Ubuntu 18.04.6 LTS   4.9.253-tegra      containerd://1.7.23-k3s2
nano97                Ready    <none>                 15d   v1.31.4+k3s1   192.168.1.97    <none>        Ubuntu 18.04.6 LTS   4.9.253-tegra      containerd://1.7.23-k3s2

Node(s) CPU architecture, OS, and Version:

 uname -a 

Linux nano90 4.9.337-tegra #1 SMP PREEMPT Mon Nov 4 23:41:41 PST 2024 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration:

1 server
8 agents

Describe the bug:

The pods running on worker nodes are unable to establish any communication with the K3s API server. This issue occurs both when trying to resolve the DNS name for the Kubernetes API (kubernetes.default.svc.cluster.local) and when attempting to connect directly to the API server using its IP address (10.43.0.1). The problem appears to be specifically tied to the worker nodes and is preventing the pods from interacting with the K3s API
Steps To Reproduce:

• Installed K3s: default configuration, without flags
  -Configure the nodes with the runtime:nvidia setting.
  -Deploy pods on the worker nodes.
  -Attempt to communicate with the K3s API server from within the pods.

Expected behavior:

Pods should be able to successfully reach the K3s API server.
Actual behavior:

The pods are unable to resolve the DNS name of the API server and fail to establish a connection.

curl -k https://kubernetes.default.svc.cluster.local
curl: (6) Could not resolve host: kubernetes.default.svc.cluster.local

Attempts to connect directly to the K3s API server's IP address (10.43.0.1) result in a connection timeout:

curl -k https://10.43.0.1
curl: (7) Failed to connect to 10.43.0.1 port 443: Connection timed out

Even basic network tests, such as ping, fail to establish a connection to the API server's IP:

ping 10.43.0.1
PING 10.43.0.1 (10.43.0.1) 56(84) bytes of data.
^C
--- 10.43.0.1 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4103ms

However, the pod is able to successfully communicate with other pods scheduled on the master node, such as the CoreDNS pod. For example, the following ping test to CoreDNS is successful:

ping 10.42.0.156
PING 10.42.0.156 (10.42.0.156) 56(84) bytes of data.
64 bytes from 10.42.0.156: icmp_seq=1 ttl=63 time=0.795 ms
64 bytes from 10.42.0.156: icmp_seq=2 ttl=63 time=1.96 ms
64 bytes from 10.42.0.156: icmp_seq=3 ttl=63 time=1.59 ms
64 bytes from 10.42.0.156: icmp_seq=4 ttl=63 time=1.84 ms
64 bytes from 10.42.0.156: icmp_seq=5 ttl=63 time=2.00 ms
^C
--- 10.42.0.156 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4009ms
rtt min/avg/max/mdev = 0.795/1.640/2.006/0.448 ms

Additional context / logs:

-The issue is isolated to the worker nodes. Pods on these nodes cannot communicate with the K3s API server, but communication works fine with other pods (like CoreDNS) on the master node.

-No specific Network Policies are in place that should restrict the communication between the pods and the API server.

-Here is some additional diagnostic information from the worker node and pod:
iptables on the worker node:

Chain INPUT (policy ACCEPT 24597 packets, 28M bytes)
 pkts bytes target     prot opt in     out     source               destination         
  104  7450 KUBE-PROXY-FIREWALL  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes load balancer firewall */
25090   28M KUBE-NODEPORTS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes health check service ports */
  104  7450 KUBE-EXTERNAL-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes externally-visible service portals */
25189   28M KUBE-FIREWALL  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   50  3000 KUBE-PROXY-FIREWALL  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes load balancer firewall */
   50  3000 KUBE-FORWARD  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding rules */
   50  3000 KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes service portals */
   50  3000 KUBE-EXTERNAL-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes externally-visible service portals */
   50  3000 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
   50  3000 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           
   50  3000 FLANNEL-FWD  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* flanneld forward */

Chain OUTPUT (policy ACCEPT 24344 packets, 4245K bytes)
 pkts bytes target     prot opt in     out     source               destination         
  438 30841 KUBE-PROXY-FIREWALL  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes load balancer firewall */
  438 30841 KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes service portals */
24880 4603K KUBE-FIREWALL  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
   50  3000 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   50  3000 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FLANNEL-FWD (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   50  3000 ACCEPT     all  --  *      *       10.42.0.0/16         0.0.0.0/0            /* flanneld forward */
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            10.42.0.0/16         /* flanneld forward */

Chain KUBE-EXTERNAL-SERVICES (2 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain KUBE-FIREWALL (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      *      !127.0.0.0/8          127.0.0.0/8          /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT

Chain KUBE-FORWARD (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain KUBE-KUBELET-CANARY (0 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain KUBE-NODEPORTS (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain KUBE-PROXY-CANARY (0 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain KUBE-PROXY-FIREWALL (3 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain KUBE-SERVICES (2 references)
 pkts bytes target     prot opt in     out     source               destination  `

iptables version on the worker node:
`sudo iptables --version
iptables v1.6.1

Services

NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP   15d

NAME             TYPE           CLUSTER-IP      EXTERNAL-IP                                                                                                             PORT(S)                      AGE
kube-dns         ClusterIP      10.43.0.10      <none>                                                                                                                  53/UDP,53/TCP,9153/TCP       15d
metrics-server   ClusterIP      10.43.250.114   <none>                                                                                                                  443/TCP                      15d
traefik          LoadBalancer   10.43.16.125    192.168.1.150,192.168.1.90,192.168.1.91,192.168.1.92,192.168.1.93,192.168.1.94,192.168.1.95,192.168.1.96,192.168.1.97   80:31322/TCP,443:31990/TCP   15d

Pods in the kube-system namespace:

NAME                                      READY   STATUS             RESTARTS          AGE
coredns-76bb48589f-2fll6                  1/1     Running            0                 2d2h
helm-install-traefik-6mz7g                0/1     Completed          0                 15d
helm-install-traefik-crd-q2njh            0/1     Completed          0                 15d
local-path-provisioner-5cf85fd84d-8mgbc   1/1     Running            0                 15d
metrics-server-5985cbc9d7-72224           1/1     Running            0                 15d
nvidia-device-plugin-daemonset-2vcsn      1/1     Running            0                 14d
nvidia-device-plugin-daemonset-4ghhr      1/1     Running            0                 14d
nvidia-device-plugin-daemonset-7hbtp      1/1     Running            0                 14d
nvidia-device-plugin-daemonset-bddmv      1/1     Running            0                 14d
nvidia-device-plugin-daemonset-ktvrw      1/1     Running            0                 14d
nvidia-device-plugin-daemonset-l6j4c      1/1     Running            0                 14d
nvidia-device-plugin-daemonset-lf582      1/1     Running            4 (36m ago)       14d
nvidia-device-plugin-daemonset-mvg6z      1/1     Running            0                 14d
svclb-traefik-63b97f81-58hg4              2/2     Running            0                 15d
svclb-traefik-63b97f81-8kgb5              2/2     Running            0                 15d
svclb-traefik-63b97f81-8wd25              2/2     Running            0                 15d
svclb-traefik-63b97f81-jxl5l              2/2     Running            0                 15d
svclb-traefik-63b97f81-qcmwx              2/2     Running            0                 15d
svclb-traefik-63b97f81-rqvjn              2/2     Running            8 (36m ago)       15d
svclb-traefik-63b97f81-sm9fk              2/2     Running            0                 15d
svclb-traefik-63b97f81-vsbz5              2/2     Running            0                 15d
svclb-traefik-63b97f81-xmssc              2/2     Running            0                 15d
traefik-57b79cf995-xlmzk                  1/1     Running            0                 15d

/etc/resolv.conf inside the test pod:

search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.43.0.10
options ndots:5

Network Interfaces on the worker node:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 16:be:05:4f:f1:eb brd ff:ff:ff:ff:ff:ff
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 48:b0:2d:d9:70:63 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.90/24 brd 192.168.1.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
4: l4tbr0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 06:e9:95:f1:b3:e9 brd ff:ff:ff:ff:ff:ff
5: rndis0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master l4tbr0 state DOWN group default qlen 1000
    link/ether 06:e9:95:f1:b3:e9 brd ff:ff:ff:ff:ff:ff
6: usb0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master l4tbr0 state DOWN group default qlen 1000
    link/ether 06:e9:95:f1:b3:eb brd ff:ff:ff:ff:ff:ff
7: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:73:a5:5c:0d brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global docker0
       valid_lft forever preferred_lft forever
8: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether c2:4d:22:a7:e9:c5 brd ff:ff:ff:ff:ff:ff
    inet 10.42.1.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
9: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 7a:d7:90:01:2b:61 brd ff:ff:ff:ff:ff:ff
    inet 10.42.1.1/24 brd 10.42.1.255 scope global cni0
       valid_lft forever preferred_lft forever
10: vethc7bcd247@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default qlen 1000
    link/ether 3e:85:ff:86:e4:af brd ff:ff:ff:ff:ff:ff link-netnsid 0
11: vethd2e5dc01@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default qlen 1000
    link/ether 6a:2f:9e:ab:f1:7a brd ff:ff:ff:ff:ff:ff link-netnsid 1
12: vethefc2ef77@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default qlen 1000
    link/ether 5a:b9:b8:a6:ae:3d brd ff:ff:ff:ff:ff:ff link-netnsid 2
13: veth71609911@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default qlen 1000
    link/ether 7a:0c:c6:be:a7:2d brd ff:ff:ff:ff:ff:ff link-netnsid 3

Routing Table on the worker node:

default via 192.168.1.254 dev eth0 proto static metric 100 
10.42.0.0/24 via 10.42.0.0 dev flannel.1 onlink 
10.42.1.0/24 dev cni0 proto kernel scope link src 10.42.1.1 
10.42.2.0/24 via 10.42.2.0 dev flannel.1 onlink 
10.42.4.0/24 via 10.42.4.0 dev flannel.1 onlink 
10.42.5.0/24 via 10.42.5.0 dev flannel.1 onlink 
10.42.6.0/24 via 10.42.6.0 dev flannel.1 onlink 
10.42.7.0/24 via 10.42.7.0 dev flannel.1 onlink 
10.42.8.0/24 via 10.42.8.0 dev flannel.1 onlink 
10.42.9.0/24 via 10.42.9.0 dev flannel.1 onlink 
169.254.0.0/16 dev eth0 scope link metric 1000 
172.18.0.0/16 dev docker0 proto kernel scope link src 172.18.0.1 linkdown 
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.90 metric 100 

[brandond]

• Make sure you have local firewalls (ufw/firewalld) disabled
• Uninstall Docker

Why are you running such an old Ubuntu + kernel on the workers?

What does your server config look like? Have you done anything weird with the bind or advertise addresses?

[DamianoSamperi]

Thank you for your response. Here are my answers to your questions:
Regarding the old Ubuntu and kernel versions on the workers:

• "I am using an older version of Ubuntu because my workers are Jetson Nano devices, which have specific compatibility requirements."

Regarding Docker:

• "I cannot uninstall Docker as it is a required dependency for the NVIDIA K8s Device Plugin. However, I did try uninstalling Docker and rebooting the node, but nothing changed after the reboot."

Regarding local firewalls:

• "The firewalls (ufw/firewalld) are already disabled."

Regarding the server config in K3s:

• "I'm not entirely sure what is meant by 'server config' in this context. If it refers to the K3s configuration, I have not made any unusual changes to it. "