AGIC is wiping out my backend pools

[justinmchase]

Describe the bug
I am enabling the app gateway ingress controller on AKS and adding Ingress objects into kubernetes and then the AGIC is wiping out all of my backend pools, settings, listeners, rules and replacing them with its own.

I need the a rule with /* to route all traffic to a cdn but some higher priority rules that match /api/* to route to AKS. Therefore I need AKS to not wipe out the existing pools.

I have tried following the instructions on how to install it manually with helm and run appgw.shared: true and setting up the target prohibitions but unfortunately the path based prohibitions dont' work in this scenario if I prohibit it from maintaining /* it also includes /api/* so then it doesn't work at all.

You seem to be lacking a not rule in the prohibitions such as:

spec:
  paths:
  - /*
  - !/api/*

There appears to be no possible configuration which will allow me have a default CDN and an aks api on the same hostname.

To Reproduce
There are probably many ways, but I am using terraform to setup an app gateway and aks and aks has this section on it:

  ingress_application_gateway {
    gateway_id = var.appgateway_id
  }

Ingress Controller details

Name:             ingress-appgw-deployment-548869cc56-w4b6x
Namespace:        kube-system
Priority:         0
Service Account:  ingress-appgw-sa
Node:             aks-system-35192060-vmss000001/10.102.0.10
Start Time:       Tue, 27 Feb 2024 18:57:37 -0600
Labels:           app=ingress-appgw
                  kubernetes.azure.com/managedby=aks
                  pod-template-hash=548869cc56
Annotations:      checksum/config: a2506bed8b059d5b313bee251646dd125aaee859b6b16914b72dc96b259b96ee
                  cluster-autoscaler.kubernetes.io/safe-to-evict: true
                  kubernetes.azure.com/metrics-scrape: true
                  prometheus.io/path: /metrics
                  prometheus.io/port: 8123
                  prometheus.io/scrape: true
                  resource-id:
                    /subscriptions/0816a7b7-daf6-4f6a-8d35-0297a9da1f73/resourceGroups/testservice-stpr99-common/providers/Microsoft.ContainerService/managedC...
Status:           Running
IP:               10.102.0.33
IPs:
  IP:           10.102.0.33
Controlled By:  ReplicaSet/ingress-appgw-deployment-548869cc56
Containers:
  ingress-appgw-container:
    Container ID:   containerd://69eb0cd344b9ba9bb4cd8b53298131ca6837d53ac38778aa6df4e80683eab435
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.2
    Image ID:       mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:eeb1d42ebfb872478d9b0b16f6936ea938d6e5eed4a59cde332b8757556a5e1f
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 27 Feb 2024 18:58:06 -0600
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     700m
      memory:  600Mi
    Requests:
      cpu:      100m
      memory:   20Mi
    Liveness:   http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:  http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      ingress-appgw-cm  ConfigMap  Optional: false
    Environment:
      AZURE_CLOUD_PROVIDER_LOCATION:  /etc/kubernetes/azure.json
      AGIC_POD_NAME:                  ingress-appgw-deployment-548869cc56-w4b6x (v1:metadata.name)
      AGIC_POD_NAMESPACE:             kube-system (v1:metadata.namespace)
      KUBERNETES_PORT_443_TCP_ADDR:   aks-3adwf6zk.hcp.centralus.azmk8s.io
      KUBERNETES_PORT:                tcp://aks-3adwf6zk.hcp.centralus.azmk8s.io:443
      KUBERNETES_PORT_443_TCP:        tcp://aks-3adwf6zk.hcp.centralus.azmk8s.io:443
      KUBERNETES_SERVICE_HOST:        aks-3adwf6zk.hcp.centralus.azmk8s.io
    Mounts:
      /etc/kubernetes/azure.json from cloud-provider-config (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qj25l (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  cloud-provider-config:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/azure.json
    HostPathType:  File
  kube-api-access-qj25l:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 CriticalAddonsOnly op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                     Age   From                       Message
  ----     ------                     ----  ----                       -------
  Warning  FailedApplyingAppGwConfig  15m   azure/application-gateway  Code="Canceled" Message="Operation was canceled." Details=[{"code":"CanceledAndSupersededDueToAnotherOperation","message":"Operation PutApplicationGatewayOperation (693b2a61-17a8-448f-a3d7-162817311e30) was canceled and superseded by operation PutApplicationGatewayOperation (33dcdf7c-18e2-49f3-87b3-bf21bd5a40f7)."}]

• Output of `kubectl logs .
  I will get this upon request, but its quite verbose...

• Any Azure support tickets associated with this issue.
  NA