OpsGenie: closing alerts automatically (using match_body)

[vitalinoborges]

Based on @jertel 's excellent tip in #337 and #115, it is possible to create a rule to monitor another rule and thereby automate the closing of alerts in OpsGenie, as in:

name: Service is back online
type: flatline
index: elastalert_status*
query_key: "rule_name"
filter:
- query:
    query_string:
      query: "rule_name:\"RULE_NAME_HERE\" AND matches:>0"
forget_keys: true
timeframe:
  minutes: 10
threshold: 1
alert:
- "opsgenie"
opsgenie:
opsgenie_addr: https://api.opsgenie.com/v2/alerts/<ALIAS NAME HERE>/close?identifierType=alias
opsgenie_key: ${OPSGENIE_API_KEY}

But this solution works great just for closing alerts that monitor a single service.

In scenarios where we have a single rule to monitor a varied set of objects (aggregated and differentiated by name, and with different aliases of OpsGenie), the above solution does not work.

In the example below, we have a single rule that triggers different alerts from different Kubernetes deployments (with the OpsGenie alias customized by the deployment name):

name: k8s Deployment Monitoring
type: frequency
index: metricbeat*
num_events: 59
timeframe:
  minutes: 10
metric_aggregation:
query_key: kubernetes.deployment.name
filter:
- query:
    query_string:
       query: "kubernetes.deployment.replicas.available: 0"
alert:
- "opsgenie"
realert:
  minutes: 60
opsgenie:
opsgenie_subject: "[Elastic Observability] {0}: Deployment Pods Available = 0.0 for at least 10 minutes on '{1}'"
opsgenie_subject_args:
- agent.name
- kubernetes.deployment.name
opsgenie_priority: "P1"
opsgenie_alias: "{kubernetes[deployment][name]}"
opsgenie_addr: https://api.opsgenie.com/v2/alerts
opsgenie_key: ${OPSGENIE_API_KEY}

I realized that all the information I need for my "Service is back online" rule to work is present in the elastalert index (instead of elastalert_status), inside the match_body object.

But strangely I can't apply filters on fields from this object (there is a similar discussion about this here, but I didn't exactly understand the limitations of match_body).

Does anyone have any ideas on how my "Service is back online" rule will be able to filter through alerts fired for different kubernetes.deployment.name's, given that the elastalert_status index does not have this information?

Ideally, something like this would work (but it doesn't):

name: Service is back online
type: flatline
index: elastalert*
query_key: "match_body.kubernetes.deployment.name"
filter:
- query:
    query_string:
      query: "rule_name:\"k8s Deployment Monitoring\" AND match_body.num_matches:>0"
forget_keys: true
timeframe:
  minutes: 10
threshold: 1
alert:
- "opsgenie"
opsgenie:
opsgenie_addr: "https://api.opsgenie.com/v2/alerts/{match_body[kubernetes][deployment][name]}/close?identifierType=alias"
opsgenie_key: ${OPSGENIE_API_KEY}

[vitalinoborges]

I found that the match_body object's fields are not "queryable", because it has mapping disabled. Consult with: GET /elastalert/_mapping

I saw an example of using match_body object fields using the "include" directive in this example:

https://github.com/jertel/elastalert2/blob/master/examples/rules/ssh-repeat-offender.yaml

I tried to reproduce as follows, but it didn't work:

name: Service is back online
type: flatline
index: elastalert
include:
- match_body.kubernetes.deployment.name
- match_body.num_matches
- rule_name
query_key: "match_body.kubernetes.deployment.name"
filter:
- query:
       query_string:
            query: "rule_name:\"k8s Deployment Monitoring\" AND match_body.num_matches:>0"
forget_keys: true
timeframe:
   minutes: 10
threshold: 1
alert:
- "opsgenie"
opsgenie:
opsgenie_addr: "https://api.opsgenie.com/v2/alerts/{match_body[kubernetes][deployment][name]}/close?identifierType=alias"
opsgenie_key: ${OPSGENIE_API_KEY}

[jertel]

This discussion has some overlap with your topic. Perhaps it is of some use to you. #527

[vitalinoborges]

This discussion helped me a lot, thank you very much.
I enabled indexing of the match_body object, built the Docker image, and now I can query the match_body fields.

But interpolation of {key} variable with OpsGenie URL is not working, like:

https://api.opsgenie.com/v2/alerts/{key}/close?identifierType=alias

I'm testing with:

name: Deployment is back online
type: flatline
index: elastalert
query_key: "match_body.kubernetes.deployment.name"
filter:
- query:
    query_string:
      query: "rule_name:\"k8s Deployment Monitoring\" AND match_body.num_matches:>0"
forget_keys: true
timeframe:
  minutes: 13
threshold: 1
alert:
- "opsgenie"
opsgenie:
opsgenie_addr: https://api.opsgenie.com/v2/alerts/{key}/close?identifierType=alias
opsgenie_key: ${OPSGENIE_API_KEY}

Do you have any idea how the variable {key} could work in opsgenie_addr?

I try:

• opsgenie_addr: https://api.opsgenie.com/v2/alerts/{key}/close?identifierType=alias
• opsgenie_addr: "https://api.opsgenie.com/v2/alerts/{key}/close?identifierType=alias"
• opsgenie_addr: https://api.opsgenie.com/v2/alerts/"{key}"/close?identifierType=alias
• opsgenie_addr: https://api.opsgenie.com/v2/alerts/{match[key]}/close?identifierType=alias
• opsgenie_addr: https://api.opsgenie.com/v2/alerts/{key[0]}/close?identifierType=alias

... but it did not work.

[nsano-rururu]

opsgenie_addr should not have the ability to replace {key}. did you read the docs?

[nsano-rururu]

https://docs.opsgenie.com/docs/alert-api

https://elastalert2.readthedocs.io/en/latest/ruletypes.html#opsgenie

[nsano-rururu]

elastalert2 is aware that the following APIs are not supported.
https://docs.opsgenie.com/docs/alert-api-continued

[vitalinoborges]

I resolved the automatic closing of resolved alerts in Opsgenie with the following procedures.

1) Enable the mapping of fields of the match_body object:

2) Create a script that receives the "alias" and "apiKey" of Opsgenie and executes a curl to close the alert.

3) Change the Dockerfile to copy the script into the image.



I used a "command" type alert to call the script. Example:

    k8s_deployment_ok: |-
      ---
      name: Deployment is back online
      type: flatline
      index: elastalert
      query_key: "match_body.kubernetes.deployment.name"
      use_terms_query: true
      doc_type: _doc
      filter:
      - query:
          query_string:
            query: "rule_name:\"k8s Deployment Monitoring\" AND match_body.num_matches:>0"
      forget_keys: true
      timeframe:
        minutes: 11
      threshold: 1
      alert:
      - "command"
      command: ["/usr/local/bin/send_alert", "%(key)s", "${OPSGENIE_API_KEY}"]

Thank you so much @jertel and @nsano-rururu.