direct routing and BPF datapath of kube-proxy replacement

[hown3d]

How to categorize this issue?

/area networking
/kind bug

What happened:
When running Cilium as a kube-proxy replacement and the eBPF datapath is chosen (will be introduced with #350) the lo device will be ignored to search for host addresses https://github.com/cilium/cilium/blob/9d631b91ad4d2c146d3decbfcfc39968764eb539/pkg/datapath/linux/devices.go#L32-L38
Running without a network overlay let's request inside containers against https://kubernetes time-out.

This currently isn not reproducible when running without overlay because bpf-masquerade get's disabled in that case:

gardener-extension-networking-cilium/charts/internal/cilium/charts/config/templates/configmap.yaml

Lines 335 to 337 in e6d1fcc

	{{- if ne .Values.global.tunnel "disabled" }}
	enable-bpf-masquerade: {{ .Values.global.enableBPFMasquerade | quote }}
	{{- end }}

Cilium will fallback to the legacy implementation of hostrouting instead of using the eBPF datapath:

$ kubectl -n kube-system logs ds/cilium
time="2024-08-07T14:01:05Z" level=info msg="BPF host routing requires enable-bpf-masquerade. Falling back to legacy host routing (enable-host-legacy-routing=true)." subsys=daemon

• tcp-dump of cilium managed node (100.83.126.209 is the service IP of kube-apiserver)

shoot--ondemand--test-worker-tyo9o-z2-6f799-cdmnr / # tcpdump -i any | grep 100.83.126.209
tcpdump: data link type LINUX_SLL2
dropped privs to pcap
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
00:16:59.303015 lxc30002a5304eb In  IP 100.64.1.234.35474 > 100.83.126.209.https: Flags [S], seq 3195040413, win 65535, options [mss 8710,sackOK,TS val 3811945852 ecr 0,nop,wscale 9], length 0: Flags [P.], seq 363384
00:16:59.303071 eth0  Out IP 100.64.1.234.35474 > 100.83.126.209.https: Flags [S], seq 3195040413, win 65535, options [mss 8710,sackOK,TS val 3811945852 ecr 0,nop,wscale 9], length 0: Flags [.], ack 362392,
00:17:00.365963 lxc30002a5304eb In  IP 100.64.1.234.35474 > 100.83.126.209.https: Flags [S], seq 3195040413, win 65535, options [mss 8710,sackOK,TS val 3811946915 ecr 0,nop,wscale 9], length 092: Flags [.], ack 6785,

• cilium-dbg output

kubectl -n kube-system exec cilium-nbtgj -- cilium-dbg statedb
 node-addresses
Defaulted container "cilium-agent" out of: cilium-agent, disable-rp-filter (init), config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
Address                     NodePort   Primary   DeviceName
10.250.2.187                true       true      eth0
100.64.0.56                 false      true      cilium_host
fe80::b407:4cff:fe39:f6fa   false      true      cilium_host

What you expected to happen:
Pods are able to access the kube-apiserver via service discovery

How to reproduce it (as minimally and precisely as possible):
Create a shoot without overlay and enable the kube-proxy replacement.
Either:

1. Add enable-bpf-masquerade: true to the cilium-config configmap in kube-system

or

2. Install cilium extension using branch of PR fix: enable-bpf-masquerade when snat values are not enabled #350

Example shoot spec to reproduce:

spec:
  kubernetes:
    kubeProxy:
      enabled: false
  networking:
    type: cilium
    providerConfig:
      apiVersion: cilium.networking.extensions.gardener.cloud/v1alpha1
      kind: NetworkConfig
      hubble:
        enabled: true
      tunnel: disabled
      ipv4NativeRoutingCIDREnabled: true
      overlay:
        enabled: false
        createPodRoutes: true

Anything else we need to know?:

Environment:

• Gardener version (if relevant): 1.96
• Extension version: built from source using base branch of PR fix: enable-bpf-masquerade when snat values are not enabled #350
• Kubernetes version (use kubectl version): 1.29.16
• Cloud provider or hardware configuration: openstack
• Others:

[hown3d]

Related issue and commit in the cilium repository.
Cilium has a hidden flag called --local-max-addr-scope which is by default to scope link (253) - 1 after v1.13.

IP addresses on a devices with scope higher than link (e.g. scope host like the apiserver-proxy creates) will be skipped.