Sane Opsgenie alerting #487
Conversation
component/slos.libsonnet
Outdated
| @@ -86,7 +86,7 @@ local generateSlothInput(name, uptime) = | |||
| }, | |||
| labels+: { | |||
| service: 'VSHN' + name, | |||
| OnCall: '{{ if eq $labels.sla "guaranteed" }}true{{ else }}false{{ end }}', | |||
| OnCall: false, | |||
There was a problem hiding this comment.
Let's remove this label completely, it doesn't serve any purpose anymore.
Also, this alone will not disable alerting to OpsGenie during office hours. This only disables routing to oncall. So whoever does ops will still get bothered by these alerts. You'll have to investigate how to disable the routing to OpsGenie. It might even need a new routing rule.
component/vshn_alerting.jsonnet
Outdated
| alert: 'vshn-' + std.asciiLower(serviceName) + '-opsgenie-ha', | ||
| // this query can be read as: if the rate of probes that are not successful is higher than 0.2 in the last 5 minutes and in the last minute, then alert | ||
| // rate works on per second basis, so 0.2 means 20% of the probes are failing, which for 5 minutes is 1 minute and for 1 minute is 12 seconds | ||
| expr: 'rate(appcat_probes_seconds_count{reason!="success", service="' + serviceName + '", ha="true", maintenance="false"}[5m]) > 0.2 and rate(appcat_probes_seconds_count{reason!="success", service="' + serviceName + '", ha="true", maintenance="false"}[1m]) > 0.2', |
There was a problem hiding this comment.
You have to remove the maintenance="false" here.
If HA instances trigger an alert during maintenance, it should be considered broken and investigated.
Also the 12seconds within 1 minute might be a bit too low for that case, as there will be failovers during the maintenance, maybe we should increase it a bit to avoid false positives?
There was a problem hiding this comment.
I just increased value for 1 minute to 45 second instead of 12, that should eliminate false positives
There was a problem hiding this comment.
also removed maitenance label
Co-authored-by: Kidswiss <[email protected]>
Co-authored-by: Kidswiss <[email protected]>
PR ensures we have new Opsgenie alerts that self resolve depending on time window, so we keep our Responsible Ops sane. It leaves old logic generated by Sloth tool, but disables forwarding alerts to Opsgenie.
Checklist
changelog.
The PR has a meaningful description that sums up the change. It will be
linked in the changelog.
bug,enhancement,documentation,change,breaking,dependencyas they show up in the changelog.