Extra nft rules from netbird causes crash loop

[timwhite]

What happened?

Sometime after starting netbird, k3s goes into a crash loop due to being unable to verify existing firewall rules.

Failed to verify rule exists in FORWARD chain due to running [/var/lib/rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17/bin/aux/iptables -t filter -C FORWARD -m comment --comment kube-router netpol - TEMCG2JMHZYE7H7T -j KUBE-ROUTER-FORWARD --wait]: exit status 3: Error: cmp sreg undef

I think it may be related to #1777, but after looking closely at the backtrace, I'm not sure if it is.
Issue was originally logged in k3s-io/k3s#11493

What did you expect to happen?

Kube network controller should better handle the error instead of crashing out.

How can we reproduce the behavior you experienced?

Start k3s node, start netbird (connected to a network), wait some amount of time until kubernetes starts to crash.

Screenshots / Architecture Diagrams / Network Topologies

System Information (please complete the following information)

• Kube-Router Version (kube-router --version): v2.2.1
• Kube-Router Parameters: unknown
• Kubernetes Version (kubectl version) : v1.29.10
• Cloud Type: onprem
• Kubernetes Deployment Type: k3s
• Kube-Router Deployment Type:
• Cluster Size: 3

Logs, other output, metrics

Dec 22 10:54:52 me5 k3s[555128]: I1222 10:54:52.715937  555128 volume_manager.go:289] "Starting Kubelet Volume Manager"
Dec 22 10:54:52 me5 k3s[555128]: I1222 10:54:52.716098  555128 dynamic_serving_content.go:135] "Starting controller" name="kubelet-server-cert-files::/var/lib/rancher/k3s/agent/serving-kubelet.crt::/var/lib/rancher/k3s/agent/serving-kubelet.key"
Dec 22 10:54:52 me5 k3s[555128]: E1222 10:54:52.719468  555128 kubelet.go:1478] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
Dec 22 10:54:52 me5 k3s[555128]: I1222 10:54:52.723440  555128 factory.go:221] Registration of the systemd container factory successfully
Dec 22 10:54:52 me5 k3s[555128]: I1222 10:54:52.724641  555128 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Dec 22 10:54:52 me5 k3s[555128]: I1222 10:54:52.726407  555128 factory.go:219] Registration of the crio container factory failed: Get "http://%2Fvar%2Frun%2Fcrio%2Fcrio.sock/info": dial unix /var/run/crio/crio.sock: connect: no such file or directory
Dec 22 10:54:52 me5 k3s[555128]: I1222 10:54:52.731858  555128 reconciler.go:26] "Reconciler: start to sync state"
Dec 22 10:54:52 me5 k3s[555128]: I1222 10:54:52.738881  555128 factory.go:221] Registration of the containerd container factory successfully
Dec 22 10:54:52 me5 k3s[555128]: I1222 10:54:52.739103  555128 kubelet_network_linux.go:50] "Initialized iptables rules." protocol="IPv4"
Dec 22 10:54:52 me5 k3s[555128]: I1222 10:54:52.741138  555128 kubelet_network_linux.go:50] "Initialized iptables rules." protocol="IPv6"
Dec 22 10:54:52 me5 k3s[555128]: I1222 10:54:52.741171  555128 status_manager.go:217] "Starting to sync pod status with apiserver"
Dec 22 10:54:52 me5 k3s[555128]: I1222 10:54:52.741193  555128 kubelet.go:2321] "Starting kubelet main sync loop"
Dec 22 10:54:52 me5 k3s[555128]: E1222 10:54:52.741250  555128 kubelet.go:2345] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Dec 22 10:54:52 me5 k3s[555128]: time="2024-12-22T10:54:52+08:00" level=info msg="Starting network policy controller version v2.2.1, built on 2024-12-05T21:29:39Z, go1.22.8"
Dec 22 10:54:52 me5 k3s[555128]: time="2024-12-22T10:54:52+08:00" level=info msg="k3s agent is up and running"
Dec 22 10:54:52 me5 k3s[555128]: I1222 10:54:52.790235  555128 network_policy_controller.go:164] Starting network policy controller
Dec 22 10:54:52 me5 systemd[1]: Started k3s-agent.service - Lightweight Kubernetes.
Dec 22 10:54:52 me5 k3s[555128]: F1222 10:54:52.796106  555128 network_policy_controller.go:400] Failed to verify rule exists in FORWARD chain due to running [/var/lib/rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17/bin/aux/iptables -t filter -C FORWARD -m comment --comment kube-router netpol - TEMCG2JMHZYE7H7T -j KUBE-ROUTER-FORWARD --wait]: exit status 3: Error: cmp sreg undef
Dec 22 10:54:52 me5 k3s[555128]: iptables v1.8.9 (nf_tables): Parsing nftables rule failed
Dec 22 10:54:52 me5 k3s[555128]: Perhaps iptables or your kernel needs to be upgraded.
Dec 22 10:54:52 me5 k3s[555128]: panic: F1222 10:54:52.796106  555128 network_policy_controller.go:400] Failed to verify rule exists in FORWARD chain due to running [/var/lib/rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17/bin/aux/iptables -t filter -C FORWARD -m comment --comment kube-router netpol - TEMCG2JMHZYE7H7T -j KUBE-ROUTER-FORWARD --wait]: exit status 3: Error: cmp sreg undef
Dec 22 10:54:52 me5 k3s[555128]: iptables v1.8.9 (nf_tables): Parsing nftables rule failed
Dec 22 10:54:52 me5 k3s[555128]: Perhaps iptables or your kernel needs to be upgraded.
Dec 22 10:54:52 me5 k3s[555128]: goroutine 686 [running]:
Dec 22 10:54:52 me5 k3s[555128]: k8s.io/klog/v2.(*loggingT).output(0xb3c6f00, 0x3, 0xc0028a97c0, 0xc00109c930, 0x1, {0x8d844fc?, 0x2?}, 0xc00189cff0?, 0x0)
Dec 22 10:54:52 me5 k3s[555128]:         /go/pkg/mod/github.com/k3s-io/klog/[email protected]/klog.go:965 +0x73d
Dec 22 10:54:52 me5 k3s[555128]: k8s.io/klog/v2.(*loggingT).printfDepth(0xb3c6f00, 0x3, 0xc0028a97c0, {0x0, 0x0}, 0x1, {0x68b8a66, 0x32}, {0xc0033e1c40, 0x2, ...})
Dec 22 10:54:52 me5 k3s[555128]:         /go/pkg/mod/github.com/k3s-io/klog/[email protected]/klog.go:767 +0x1f0
Dec 22 10:54:52 me5 k3s[555128]: k8s.io/klog/v2.(*loggingT).printf(...)
Dec 22 10:54:52 me5 k3s[555128]:         /go/pkg/mod/github.com/k3s-io/klog/[email protected]/klog.go:744
Dec 22 10:54:52 me5 k3s[555128]: k8s.io/klog/v2.Fatalf(...)
Dec 22 10:54:52 me5 k3s[555128]:         /go/pkg/mod/github.com/k3s-io/klog/[email protected]/klog.go:1655
Dec 22 10:54:52 me5 k3s[555128]: github.com/cloudnativelabs/kube-router/v2/pkg/controllers/netpol.(*NetworkPolicyController).ensureTopLevelChains.func2({0x779ca60, 0xc001564550}, {0x675dfb2, 0x7}, {0xc001317aa0, 0x6, 0x6}, {0xc0028608c0, 0x10}, 0x1)
Dec 22 10:54:52 me5 k3s[555128]:         /go/pkg/mod/github.com/k3s-io/kube-router/[email protected]/pkg/controllers/netpol/network_policy_controller.go:400 +0x1b2
Dec 22 10:54:52 me5 k3s[555128]: github.com/cloudnativelabs/kube-router/v2/pkg/controllers/netpol.(*NetworkPolicyController).ensureTopLevelChains(0xc00332d560)
Dec 22 10:54:52 me5 k3s[555128]:         /go/pkg/mod/github.com/k3s-io/kube-router/[email protected]/pkg/controllers/netpol/network_policy_controller.go:467 +0x1be9
Dec 22 10:54:52 me5 k3s[555128]: github.com/cloudnativelabs/kube-router/v2/pkg/controllers/netpol.(*NetworkPolicyController).Run(0xc00332d560, 0xc00287c120, 0xc0000d6360, 0xc002c3afd0)
Dec 22 10:54:52 me5 k3s[555128]:         /go/pkg/mod/github.com/k3s-io/kube-router/[email protected]/pkg/controllers/netpol/network_policy_controller.go:168 +0x171
Dec 22 10:54:52 me5 k3s[555128]: created by github.com/k3s-io/k3s/pkg/agent/netpol.Run in goroutine 1
Dec 22 10:54:52 me5 k3s[555128]:         /go/src/github.com/k3s-io/k3s/pkg/agent/netpol/netpol.go:184 +0xe34
Dec 22 10:54:52 me5 systemd[1]: k3s-agent.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Dec 22 10:54:52 me5 systemd[1]: k3s-agent.service: Failed with result 'exit-code'.
Dec 22 10:54:52 me5 systemd[1]: k3s-agent.service: Unit process 146212 (containerd-shim) remains running after unit stopped.

Additional context

Extra rules added by netbird when kube is running first are in chain FORWARD and they are:

               oifname "wt0" ct state established,related counter packets 0 bytes 0 accept
               iifname "wt0" counter packets 0 bytes 0 accept

Netbird is changing nft rules, however according to the output of nft list table ip filter after starting netbird (before k3s), it's being managed by iptables-nft, so I'm not sure where the incompatibility is coming from.

[aauren]

I'm going to guess its a problem with the user space tooling being on different versions. The version of iptables that we currently bake into the alpine image is v1.8.9 (although I'm not sure if that's what k3s uses or if they use a different container base). If you are able to exec into the image and run iptables --version you should be able to see it.

As a helpful troubleshooting step, can you please change your host's iptables userspace to the same one that k3s is running and then see if this problem still occurs?

Similar to what I said here: #1777 (comment) the upstream netfilter project recently hasn't been ensuring backwards compatibility in the userspace.

[aauren]

It looks like iptables-1.8.11 fixes the issue with checking iptables rules that was introduced in iptables-1.8.10 and kept us on a legacy iptables userspace in the kube-router container (1.8.9).

My hope is that once Alpine releases a version that includes the iptables-1.8.11 userspace that we can use that and it should hopefully make it more resilient to host userspace tooling versions, since it should hopefully be more compatible with newer userspace version.

While we wait for Alpine to release, I've built my own iptables 1.8.11 packages for Alpine and included them in a PR build here: #1790

If the machine that you're testing doesn't have access to sensitive data, and you're willing to trust custom iptables binaries made by the project, and your cluster is using AMD64 architecture machines, you can give cloudnativelabs/kube-router-git:PR-1790 a try: https://hub.docker.com/layers/cloudnativelabs/kube-router-git/PR-1790/images/sha256-cbf606605c9e5d9111c952e96d337074e83a60e495f30cac74c5d85fd27252f2

[timwhite]

I'm going to guess its a problem with the user space tooling being on different versions. The version of iptables that we currently bake into the alpine image is v1.8.9 (although I'm not sure if that's what k3s uses or if they use a different container base). If you are able to exec into the image and run iptables --version you should be able to see it.

As a helpful troubleshooting step, can you please change your host's iptables userspace to the same one that k3s is running and then see if this problem still occurs?

I'm a bit lost with working out the iptables version in the k3s implementation of kube-router, so I think I'll bounce back to them to try and work that out.

Something I missed in my original report was that there is no userspace iptables command on the host system. So I'm guessing netbird is doing something weird as well, given that after a nft flush ruleset and then restarting netbird, nft states # Warning: table ip filter is managed by iptables-nft, do not touch! even before k3s has done anything.

Hopefully the k3s project can help me confirm the iptables version being used, and what might be needed to test with your updated image. Thanks

[timwhite]

From what I can see, k3s is using iptables 1.8.9 still. I've bumped the upstream to see what it would take to change that. If I install userspace iptables, it will also be 1.8.9. So I might just try installing the userspace iptables as well, as then maybe netbird and k3s will use that instead.

[aauren]

So, it looks like netbird uses (https://github.com/netbirdio/netbird/blob/main/client/firewall/nftables/rule_linux.go) a Google Go implementation to manipulate nftables (https://github.com/google/nftables).

Unlike iptables, nftables as a published user-space API and you don't have to use the project's official user-space library / binary in order to manipulate them.

I can certainly understand why netbird would want to use the Google library given that its a pure go implementation and it creates a consistent user experience, meaning that they don't have to be beholden to whatever library versions people are using on the host system. It makes the binary that they produce completely self-sufficient, kind of like a container without requiring people to run netbird in a container.

However, to me it is a pretty big red flag and it means that kube-router and other tools may never be copacetic with netbird. If the upstream netfilter project has such a hard time ensuring backwards compatibility with different user spaces, I would have very little hope that all of the unofficial Google Library rules that it writes will be problem free. I wouldn't even expect them to be fully compatible with all versions of the official nft binary, although netbird is much more likely to test against those and try to fix reports of breakages there.

While this is just a guess, I think that it is most likely that the problems you're experiencing have to do with netbird via the Google library writing nft rules that are not fully backward compliant with the official iptables legacy binary that is distributed by the netfilter project. Best bet would be to file this issue with the netbird project and hope that someone on the project is either able to work around the library or get someone it Google to fix the incompatibility. Although on the site for the Google library they say they don't accept GitHub issues and that the library isn't official supported by Google so mileage may vary.

One last thing I'll throw out, if the problem that you happen to be experiencing right now in the incompatibility between netbird and kube-router, is related to kube-router using older versions of the iptables user-space, it is possible that the release I pointed you to may help with that. It is still possible that this specific bug is because the Google nftables library is using a new feature that the legacy iptables user-space cannot decode. If this is the case, updating the user-space like I did in the linked kube-router container, would fix this specific issue.