Rule throws an error is disabled

[phlegx]

Hi there!

I try to monitor my load on my machine with the following rule config (I use Terraform hence some values are templated):

name: "[${project_name}][${environment}] Load on ${hostname} ${severity}"
description: Monitors Load on the bare metal hosts in ${environment} environment
type: metric_aggregation
index: ${project_shortcut}-metricbeat-${environment}*
metric_agg_key: system.load.1
metric_agg_type: avg
query_key: system.load.1

max_threshold: ${max_threshold}
filter:
- term:
    metricset.name: load

alert_subject: "[${project_name}][${environment}] System load ${severity} - {0} percent in use on server ${hostname}"
alert_subject_args:
- system.load.1

realert:
  minutes: 1
alert:
  - pagerduty

pagerduty_service_key: "${pagerduty_integration_key}"
pagerduty_client_name: "Elastalert"
pagerduty_event_type: "trigger"

pagerduty_api_version: "v2"
pagerduty_v2_payload_severity: "${severity}"
pagerduty_v2_payload_source: host.name
pagerduty_incident_key: "${project_shortcut}-${environment}-systemload-${hostname}-${severity}"

Elastalert then, on when the threshold is reached, sends indeed an alert to Pagerduty. However I also get an error right afterwards and the rule seems to be disabled. if I resolve the Pagerduty alert manually in pagerduty I do not get a second alert anymore. Here is the verbose output:

INFO:elastalert:Starting up
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.999937 seconds
INFO:elastalert:Background configuration change check run at 2021-05-19 19:45 +0000
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.999878 seconds
ERROR:elastalert:Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/elastalert/elastalert.py", line 1506, in alert
    return self.send_alert(matches, rule, alert_time=alert_time, retried=retried)
  File "/usr/local/lib/python3.9/site-packages/elastalert/elastalert.py", line 1600, in send_alert
    alert.alert(matches)
  File "/usr/local/lib/python3.9/site-packages/elastalert/alerts.py", line 1365, in alert
    body = self.create_alert_body(matches)
  File "/usr/local/lib/python3.9/site-packages/elastalert/alerts.py", line 261, in create_alert_body
    body += str(BasicMatchString(self.rule, match))
  File "/usr/local/lib/python3.9/site-packages/elastalert/alerts.py", line 151, in __str__
    self._add_rule_text()
  File "/usr/local/lib/python3.9/site-packages/elastalert/alerts.py", line 103, in _add_rule_text
    self.text += self.rule['type'].get_match_str(self.match)
  File "/usr/local/lib/python3.9/site-packages/elastalert/ruletypes.py", line 1084, in get_match_str
    match[self.metric_key],
KeyError: 'metric_system.load.1_avg'

ERROR:elastalert:Uncaught exception running rule [PROJECTNAME][staging] Load on staging1 critical: 'metric_system.load.1_avg'
INFO:elastalert:Rule [PROJECTNAME][staging] Load on staging1 critical disabled
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2021-05-19 19:45 +0000
INFO:elastalert:Trigger sent to PagerDuty
INFO:elastalert:Ran [PROJECTNAME][staging] Load on staging1 critical from 2021-05-19 19:40 +0000 to 2021-05-19 19:45 +0000: 30 query hits (0 already seen), 1 matches, 1 alerts sent
INFO:elastalert:[PROJECTNAME][staging] Load on staging1 critical range 300
INFO:elastalert:Background configuration change check run at 2021-05-19 19:45 +0000
INFO:elastalert:Disabled rules are: ['[PROJECTNAME][staging] Load on staging1 critical']
INFO:elastalert:Sleeping for 9.999863 seconds

Any ideas? Is my rule config correct? If not, does anyone know what I'm doing wrong?

Does this INFO:elastalert:Disabled rules are mean, that the rule is disabled forever or just for a timeframe?

[jertel]

I'm not seeing anything that looks wrong with what you have. Enabling debug might help show more information to better point us to the problem.

The behavior of rules being disabled upon encountered an unhandled error is documented here: https://elastalert2.readthedocs.io/en/latest/elastalert.html?highlight=disable_rules_on_error

[nsano-rururu]

@phlegx

I think it's a good idea to run the rule with --es_debug_trace. See the yelp/elastalert issue below for how to use it.
Yelp/elastalert#2701