[API Gateway] no access to consul api gateway pod health check probe endpoint /ready on https (k8s) for Azure AGIC health checks

[pawellegowski89]

Overview of the Issue

I have consul on kubernetes as AKS (azure kubernetes service). In addition, I use Azure AGIC (Application Gateway Ingress Controller) I have implemented service mesh. I also use cosnul api gateway. Traffic pattern:

Public -> AAGW (Azure Aplication Gateway) -> MyIngress -> Cosnsul API Gateway -> HTTPRoute ->(Intention)-> myservice app

Almost everything is ok, however, one error occurs:

Consul api gateway I implement with CRD, kind: Gateway, gatewayClass: consul, listens on port 443 and terminates SSL.
Because of this, I have the following annotations in Ingress:

appgw.ingress.kubernetes.io/backend-hostname: myhost
appgw.ingress.kubernetes.io/backend-protocol: https
kubernetes.io/ingress.class: azure/application-gateway

For Azure AGIC (Application Gateway Ingress Controller), it sends a regular health check to the Api consul gateway (which is in the service mesh). Unfortunately, the consul API gateway itself listens on https, and his pod endpoind for rediness probe listens on HTTP.

I can't change in ingress: appgw.ingress.kubernetes.io/backend-protocol: https
to http because consul api gateway itself terminates ssl so it has to listen on 443.

Annotations available in AGIC:

appgw.ingress.kubernetes.io/health-probe-port
appgw.ingress.kubernetes.io/health-probe-path 

Where to point out port and path -> for consul api gatewey are:
/ready and port 21000

Unfortunately, despite their setting, it is not for ingress to set up a separate backend-protocol just for the health check probe.

Another problem is that the svc that is created automatically by the consul for api gateway only exposes port 443, and does not expose the port from the health check probe given in api gateway pod.

The question is:

How to configure cosnsul api gateway and AGIC correctly?
I mean how enable traffic for AGIC healt check queries to healt check /ready endpoint on port 21000 (cosnul api gateway pod)?

---

Reproduction Steps

Steps to reproduce this issue, eg:

1. Deploy consul on k8s (AKS)
2. Add CR with kind: Gateway with listener on 443 on / (PrefixPath), with myhost
3. Deploy AGIC on k8s (AKS)
4. Add ingress for myhost on 443 with myhost on / (PrefixPath) and appgw.ingress.kubernetes.io/backend-protocol: https
5. Check on Azure Portal that ingress -> will be unhealthy due to inability to send healt check to pod consul api gateway

Consul info for both Client and Server

consul 1.19.2
consul-k8s 1.5.3

Operating system and Environment details

k8s - Azure Kubernetes Service v1.29.2