linkerd-control-plane pods fail with linkerd-cni and proxy.nativeSidecar enabled

[krzysztof-mitus]

What is the issue?

Helm chart linkerd-control-plane install fails when having cniEnabled=true and proxy.nativeSidecar=true.
Both linkerd-destination and linkerd-proxy-injector fail on linkerd-network-validator container in CrashLoopBackOff state with the following log:

2024-04-04T08:59:17.257953Z  INFO linkerd_network_validator: Listening for connections on 0.0.0.0:4140
2024-04-04T08:59:17.257986Z DEBUG linkerd_network_validator: token="PqBwBvtO5lWbgVvHjaZ0MQQTqqZwjmkfQPtxoK4iysvQ5KG0zyDpEWQtpoZXfLU\n"
2024-04-04T08:59:17.257993Z  INFO linkerd_network_validator: Connecting to 1.1.1.1:20001
2024-04-04T08:59:27.259562Z ERROR linkerd_network_validator: Failed to validate networking configuration. Please ensure iptables rules are rewriting traffic as expected. timeout=10s

How can it be reproduced?

1. On EKS 1.29 install helm charts:

• linkerd-crds 2024.3.5
• linkerd2-cni 2024.3.5
• linkerd-control-plane 2024.3.5

2. linkerd-control-plane helm values:

• cniEnabled: true
• proxy.nativeSidecar: true

Logs, error output, etc

see above

output of linkerd check -o short

× control plane pods are ready

Environment

• EKS 1.29
• Bottlerocket OS 1.19.3 (aws-k8s-1.29)

Possible solution

Issue probably caused by the code:
https://github.com/linkerd/linkerd2-proxy-init/blob/main/cni-plugin/main.go#L196

See slack

Additional context

Control plane pods are running when proxy.nativeSidecar: false. Enabling the feature causes pods failure.

Would you like to work on fixing this bug?

None

[mateiidavid]

@krzysztof-mitus thanks for raising this! So, if I understand correctly, the problem is that when you use nativeSidecar you still get the init container when you shouldn't, since it's supposed to be disabled through cniEnabled?

[alpeb]

I think @krzysztof-mitus's pointer to the code is spot on. The issue is the cni plugin is looking for the proxy in the list of containers, but in this case it should look in the init containers. Sounds like an easy fix, will try to push something ASAP.

[alpeb]

I've pushed a fix to linkerd/linkerd2-proxy-init#362.
@krzysztof-mitus pending a linkerd-cni release, you can give this a try with the image I published ghcr.io/alpeb/cni-plugin:v1.4.1 that contains that fix.

[krzysztof-mitus]

thanks Alejandro. I will test it

[msuszko-vertex]

I can confirm linkerd/linkerd2-proxy-init#362 does help. Linkerd 2024.4.1 with proxy.nativeSidecar: true and cniEnabled: true runs linkerd-network-validator initContainer successfully.

[krzysztof-mitus]

thanks for quick resolution and test