OPNET-629: Mark haproxy unhealthy if no healthy backends #4767
Conversation
Previously we avoided doing this because of potential issues in unhealthy clusters where backends were flapping and we didn't want to trigger failovers. However, given the nature of the firewall rule monitor check that approach was not effective anyway and allowing HAProxy to report its own status to the monitor is much more robust than relying on API calls being routed correctly when API rollouts are happening. This is being implemented as a separate monitor endpoint because we don't want the Kubelet liveness probes to fail just because there are no backends (which is an expected state in early cluster deployment). That would trigger unnecessary crash loops.
openshift-ci-robot
added
the
jira/valid-reference
|
@cybertron: This pull request references OPNET-629 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
This is the runtimecfg change corresponding to openshift/machine-config-operator#4767 which switches the monitor call to the HAProxy endpoing rather than call through to the API.
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: cybertron The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
In order to improve the robustness of the on-prem HAProxy instance, we have added a second healthcheck port in openshift/machine-config-operator#4767 This corresponds to the existing 9444 port, but because the surrounding ports were already in use I moved it an even 10 away.
|
@cybertron: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Previously we avoided doing this because of potential issues in unhealthy clusters where backends were flapping and we didn't want to trigger failovers. However, given the nature of the firewall rule monitor check that approach was not effective anyway and allowing HAProxy to report its own status to the monitor is much more robust than relying on API calls being routed correctly when API rollouts are happening.
This is being implemented as a separate monitor endpoint because we don't want the Kubelet liveness probes to fail just because there are no backends (which is an expected state in early cluster deployment). That would trigger unnecessary crash loops.
- What I did
- How to verify it
- Description for the changelog