Resiliency: Traffic ingress health checks should be configured

[niallthomson]

Is your idea request related to a problem that you've solved? Please describe.
Customer operating workloads on EKS had an operational issue affecting worker nodes and the pods scheduled on them. Although pod health checks had been configured there was not appropriate health checks configured on the ingress routing traffic to these pods, resulting in persistent 504 errors until the pods were evicted (5+ minutes). If health checks had been configured at the traffic ingress level then the pods would have been marked as unhealthy targets and removed from the pool.

Describe the best practice
When using an Ingress or Service of type LoadBalancer appropriate health checks should be configured so that the load balancer can perform its own checks of the health of its targets. For example with the AWS Load Balancer Controller the health check annotations for Ingress or Service can be leveraged to ensure the respective ALB or NLB is perform health checks of the targets independent of the Kubernetes probes.

Describe alternatives you've considered
N/A

Additional context
N/A