Possible issue with is_enabled and elastalert-test-rule command

[invisibleninja06]

trying to test a rule before I enable it so I set is_enabled to false and when i run elastalert-test-rule against the rule im getting

Traceback (most recent call last):
  File "/usr/local/bin/elastalert-test-rule", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elastalert/test_rule.py", line 500, in main
    test_instance.run_rule_test()
  File "/usr/local/lib/python3.11/site-packages/elastalert/test_rule.py", line 490, in run_rule_test
    self.run_elastalert(rule_yaml, conf)
  File "/usr/local/lib/python3.11/site-packages/elastalert/test_rule.py", line 409, in run_elastalert
    client.run_rule(rule, self.endtime, self.starttime)
  File "/usr/local/lib/python3.11/site-packages/elastalert/elastalert.py", line 822, in run_rule
    for x in range(len(rule['agg_matches'])):
                       ~~~~^^^^^^^^^^^^^^^
KeyError: 'agg_matches'

I took a rule that i know work and passes with flying colors with elastalert-test-rule and than added is_enabled: false to it and sure enough it now also fails with same exact error.

Looking to see if im overlooking anything but seems like an issue, also noticed that the is_enabled is not defined anywhere in the schema file either.

I see one other discussion that mentioned having the KeyError: 'agg_matches' but it was not in the context of testing and as I outlined its easily reproducible on my end.

[jertel]

The is_enabled flag is used by ElastAlert 2's initialization stage to separate rules into an enabled list, and a disabled list. Rules in the disabled list do not get fully initialized. Later during the scheduling loop, the enabled list of rules get run on each scheduled cycle.

elastalert-test-rule uses that same initialization logic, which detects that the rule is disabled. However, since you are telling the script "RUN THIS RULE!" via the command line arguments, it bypasses the scheduling logic and immediately runs the specified rule. Yet the rule was not properly initialized because it's marked as a disabled rule.

So you have an awkward scenario here, where your telling a helper tool to do something that doesn't make sense. Ideally, ElastAlert 2 would show a more graceful error that you're doing something that isn't possible, but currently it aborts with a stack trace. If you're interested in submitting a PR to make it more gracefully exit, you are welcome to do so. The contributing guidelines document specifies requirements for submissions.

[invisibleninja06]

Thanks for breaking it down, this is kind of what I assumed was happening in the background.

I would argue it does make sense that a testing tool that I invoke specifically on a rule should not care whether the rule is enabled or not. Our rules get synced with git and I don't want my rule to run before I test them. Sure I can make a whole other folder to place rules for testing that is not picked up by elastalert so that I can avoid this but from the doc it appeared quite feasible using the is_enabled.

Regardless ill make an issue on the project for a better error and to amend the documentation (since it doesn't mention that this parameter will break the tool)

Would it be pointless to suggest a --ignore_is_enabled parameter for the elastalert-test-rule that way disabled rules could be tested without breaking the expected behavior that some may expect?

[jertel]

If the test tool ignored the is_enabled=false property by default it could make it difficult for users to figure out why the rule alerts during testing but not during production. So your idea of an --ignore_is_enabled flag would be a way to solve this for your use case without misleading others that aren't expecting that behavior. I'd be onboard with that change.