Cannot run kubectl get logs - failed to verify certificate: x509

[brpaz]

Hello.

I have a server with the following network interfaces:

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:a3:d7:92 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname ens5
    inet 192.168.121.39/24 brd 192.168.121.255 scope global dynamic eth0
       valid_lft 2350sec preferred_lft 2350sec
    inet6 fe80::5054:ff:fea3:d792/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:a0:ad:69 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    altname ens6
    inet 192.168.55.10/24 brd 192.168.55.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fea0:ad69/64 scope link 
       valid_lft forever preferred_lft forever

This is a Vagrant machine, so all IPs are private range, but the same behavior happens in a proper cloud environment with a private and public network interface.

In this scenario the 192.168.55.10 is the internal network one, and the 192.168.121.39 is the external.

I want to configure my k3s server to use the private network, so I set --node-ip=192.168.55.10`.

Everything works fine the cluster is working and I can deploy workloads.

But when I run kubectl logs I always get the following error:

Error from server: Get "https://192.168.55.10:10250/containerLogs/default/nginx-deployment-5bb85d69d8-nmzh9/nginx-container?follow=true": tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, 192.168.121.39, not 192.168.55.10

All other kubectl commands work, only logs is giving an error.

I am very confused with this. Why only the logs command giving problems?

From the error message, It complains the the internal iP is not configured for the certificate., despite being what is set in node-ip.

I tried adding the private IP do tls-san, but it´s the same.

I got this in the logs. It seems to be registering all the IPs correctly.

Jul 06 12:15:36 control-plane k3s[7857]: time="2024-07-06T12:15:36Z" level=info msg="Updating TLS secret for kube-system/k3s-serving (count: 11): map[listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-192.168.121.39:192.168.121.39 listener.cattle.io/cn-192.168.55.10:192.168.55.10 listener.cattle.io/cn-

It only works, if I set node-ip to the public-ip, which is not ideal, since I want to use the private network, for internal cluster communications.

Is there any other configuration I need to set, like play with advertise-addr or something?

Really appreciate your help.

[brandond]

The kubelet should respect the configured node IP. Can you show the output of kubectl get node -o yaml?

Did you perhaps add the --node-ip flag after the fact? You may need to use kubectl delete node to delete the node, and then restart it to get it to rejoin, so that the address can be properly set.