failing network policy to enable dns traffic in OpenShift

[enrico2828]

We are runnning OpenShift 4.10 on baremetal servers with the Cisco ACI CNI plugin version 5.2.3.4 and ACI 5.2(5c). When we apply a NetworkPolicy.networking.k8s.io/v1 of type egress, we cannot get the DNS service working as expected.

HOW TO REPRODUCE
Deploy the yamls added to this issue. This will deploy:

• nginx instance in namespace na-nettest-nginx, exposed on a service on port 8080
• pod with curl in namespace na-nettest-curlclient
• egress policies in namespace na-nettest-curlclient that will
• default deny all egress traffic
• allow egress traffice to namespace na-nettest-nginx for the nginx pod on port 8080
• allow egress traffic to namespace openshift-dns on port 5353 and 53 (the dns pods are running 5353, actually we think this should be sufficient)

> oc describe netpol allow-curler
Name: allow-curler
Namespace: na-nettest-curlclient
Created on: 2023-01-25 16:29:42 +0100 CET
Labels: <none>
Annotations: <none>
Spec:
PodSelector: app=curl-green
Not affecting ingress traffic
Allowing egress traffic:
To Port: 8080/TCP
To:
NamespaceSelector: kubernetes.io/metadata.name=na-nettest-nginx
PodSelector: app=nginx-green
----------
To Port: 5353/TCP
To Port: 5353/UDP
To Port: 53/TCP
To Port: 53/UDP
To:
NamespaceSelector: kubernetes.io/metadata.name=openshift-dns
Policy Types: Egress

The curl pod will continuously curl nginx-green.na-nettest-nginx.svc.cluster.local:8080.

ACTUAL RESULT
"curl: (28) Resolving timed out after 5000 milliseconds" Dns traffic is not working.

Drop logs on curl pod reveal:
Warning Acc-SEC_GROUP_OUT_TABLE MISS(Security Drop) 17s (x2 over 2m22s) aci-containers-host IPv4 packet from na-nettest-curlclient/curl-green-6b9686457f-vzmlk to 10.122.0.10 was dropped

This is the IP address of the Cluster DNS service.

> oc get svc -n openshift-dns
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dns-default ClusterIP 10.122.0.10 <none> 53/UDP,53/TCP,9154/TCP 89d

In the APIC in the HostProtection profile we see that there are no rules created that target the DNS service.

EXPECTED RESULT
The curl is expected to resolve the address and output the nginx result, http code 200.

It is expected than when we target the namespace of the DNS pods and the ports of the pods, that communication to the DNS service is permitted.

IS THERE A WORKAROUND?
We found that if we explicitly add the ip address of the DNS service as ipBlock to the egress, DNS traffic is working. However, we do not see that this is necessary with other network plugins. As stated earlier, targeting the pods should enable the services as well.

---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-curler
  namespace: na-nettest-curlclient
spec:
  policyTypes:
  - Egress
  podSelector: 
    matchLabels:
      app: curl-green
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: na-nettest-nginx
      podSelector:
        matchLabels:
          app: nginx-green
    ports: 
    - protocol: TCP
      port: 8080
  - to:
    - ipBlock:
        cidr: 10.122.0.10/32
    ports: 
    - protocol: TCP
      port: 53
    - protocol: UDP
      port: 53

What is strange, if we test with the workaround, the egress policy to target the nginx pods for the curl is working flawlessly, even if we have the ports of service and pod deviating. We see in the APIC that a rule is added for the IP address of the nginx service. We really wonder why the same principle is not working for the DNS service.

KUBERNETES RESSOURCES

---
apiVersion: v1
kind: Namespace
metadata:
  name: na-nettest-nginx
---
apiVersion: v1
kind: Namespace
metadata:
  name: na-nettest-curlclient
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx-green
  name: nginx-green
  namespace: na-nettest-nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-green
  strategy: {}
  template:
    metadata:
      labels:
        app: nginx-green
    spec:
      containers:
      - image: nginxinc/nginx-unprivileged:1.23-alpine-slim
        name: nginx-unprivileged
        resources: {}
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx-green
  name: nginx-green
  namespace: na-nettest-nginx
spec:
  ports:
  - name: "8080"
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: nginx-green
  type: ClusterIP
status:
  loadBalancer: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: curl-green
    test: nginx
  name: curl-green
  namespace: na-nettest-curlclient
spec:
  replicas: 1
  selector:
    matchLabels:
      app: curl-green
  strategy: {}
  template:
    metadata:
      labels:
        app: curl-green
    spec:
      containers:
      - image: registry.access.redhat.com/ubi8@sha256:323bb3abab06523d595d133fe82c40c13aea0242445c30c9842d34c81c221dea
        name: curl-green
        command: ["/bin/bash"]
        args:
        - -c 
        - 'while true; do date && printf "\n" && curl -I --connect-timeout 5 http://nginx-green.na-nettest-nginx.svc.cluster.local:8080; sleep 5 && printf "\n\n\n"; done'
        ressources: {}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: na-nettest-curlclient
spec:
  podSelector: {}
  policyTypes:
  - Egress
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-curler
  namespace: na-nettest-curlclient
spec:
  policyTypes:
  - Egress
  podSelector: 
    matchLabels:
      app: curl-green
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: na-nettest-nginx
      podSelector:
        matchLabels:
          app: nginx-green
    ports: 
    - protocol: TCP
      port: 8080
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: openshift-dns
    ports: 
    - protocol: TCP
      port: 5353
    - protocol: UDP
      port: 5353
    - protocol: TCP
      port: 53
    - protocol: UDP
      port: 53