Test rule returns hits, but running the actual rule doesn't work?

[RLStepper]

I have a rule that I'm running in elastalert-test-rule, and it returns the hits appropriately as expected. But, when I run elastalert, all I see is:

2023-02-16 19:05:01,748     INFO           elastalert 1 rules loaded
2023-02-16 19:05:01,752     INFO           elastalert Starting up
[ec2-user@ip-xx-xx-xx-xx elastalert2]$ 

I've run other rules that have gotten 0 hits, but still show the progressive info (and write to the writeback index successfully in my cluster) after "elastalert Starting up" instead of just jumping back to a fresh command line immediately. I've tried to use es_debug_trace, but nothing is in my trace.log after doing that - it's simply blank. Any help with this would be greatly appreciated, as I've exhausted literally every search engine resource related to elastalert at this point.

In case it's needed, here are the config and rule files being referenced:

# This is the folder that contains the rule yaml files
# This can also be a list of directories
# Any .yaml file will be loaded as a rule
rules_folder: /home/ec2-user/elastalert2/elastalert2/examples/rules

# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  minutes: 1

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 15

# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: removed**

# The Elasticsearch port
es_port: 443

# Connection timeout
es_conn_timeout: 120

# The AWS region to use. Set this when using AWS-managed elasticsearch
aws_region: removed**

# The AWS profile to use. Use this if you are using an aws-cli profile.
# See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
# for details
# profile: jumpbox

# Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch

# Optional prefix for statsd metrics
#statsd_instance_tag: elastalert

# Optional statsd host
#statsd_host: dogstatsd

# Connect with TLS to Elasticsearch
use_ssl: True

# Verify TLS certificates
verify_certs: True

# Show TLS or certificate related warnings
#ssl_show_warn: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See https://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for Elasticsearch
es_username: removed**
es_password: removed**

# Use SSL authentication with client certificates client_cert must be
# a pem file containing both cert and key for client
#ca_certs: /path/to/cacert.pem
#client_cert: /path/to/client_cert.pem
#client_key: /path/to/client_key.key

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

# Optional timestamp format.
# ElastAlert will print timestamps in alert messages and in log messages using this format.
#custom_pretty_ts_format: '%Y-%m-%d %H:%M'

# Custom logging configuration
# If you want to setup your own logging configuration to log into
# files as well or to Logstash and/or modify log levels, use
# the configuration below and adjust to your needs.
# Note: if you run ElastAlert with --verbose/--debug, the log level of
# the "elastalert" logger is changed to INFO, if not already INFO/DEBUG.
logging:
  version: 1
  incremental: false
  disable_existing_loggers: false
  formatters:
    logline:
      format: '%(asctime)s %(levelname)+8s %(name)+20s %(message)s'

  handlers:
    console:
      class: logging.StreamHandler
      formatter: logline
      level: DEBUG
      stream: ext://sys.stderr

    file:
      class : logging.FileHandler
      formatter: logline
      level: DEBUG
      filename: elastalert.log

  loggers:
    elastalert:
      level: WARN
      handlers: []
      propagate: true

    elasticsearch:
      level: WARN
      handlers: []
      propagate: true

    elasticsearch.trace:
      level: WARN
      handlers: []
      propagate: true

    '':  # root logger
      level: WARN
      handlers:
        - console
        - file
      propagate: false

Rule File:

name: testrule
type: any
index: log-*
filter:
- term:
    event.action: REJECT
- term:
    source.address: removed***
- term:
    destination.ip: removed***
alert:
- debug
timestamp_field: "@timestamp"

This rule file was made specifically for troubleshooting this issue and will 100% get multiple hits (worked as expected using elastalert-test-rule).

[jertel]

I don't think it's a rule problem. Yours looks fine. How are you starting ElastAlert 2?

[RLStepper]

Thanks Jertel, I'll give it a shot and report back my findings.

[RLStepper]

Well, I've run into nothing but trouble regardless of the method in which I run Elastalert2. For better context, my ES cluster is hosted in AWS (OpenSearch 2.3), and I am attempting to run Elastalert2 using an EC2 with all of the required networking configurations for it to communicate with my cluster (VPC, security groups, rules, roles, etc.) I have a few questions that may help me figure this out, if you have time @jertel

• Is there a particular way you would recommend going about this?
• Is elastalert run from inside my container? I would think no, since the run command has "always restart" in it.
• I have no need to install Elasticsearch on my EC2, as I'm using OpenSearch, so there is no ES container running in the same network (es_default) as elastalert (though my EC2 and OpenSearch are in the same VPC, networking is g2g, etc). Have you, or do you know anyone, who was able to get elastalert2 working in a similar scenario?

Thanks in advance. With AWS only having OpenSearch 2.3, I am unable to utilize essentially the same functionality that elastalert2 provides (OS2.4 has the Security Plugin), so I would really like to use elastalert2 to gain that functionality. Just a bit of an explanation as to why I'm still at it.

[jertel]

I recommend using Docker. That way it works the same on your environment as it does on mine. There is no reason why ElastAlert 2 would not run in EC2. It is a common deployment method and I've never run into problems doing so. In fact, I have automated tests that deploy several new EC2 instances with ElastAlert 2 on them, every night.

If you use Docker than yes, ElastAlert 2 runs from the container. If you don't understand how Docker works, it can be thought of as a virtual filesystem, yet the process still runs on the host. So all file and lib dependencies are isolated from the host file system.

There is no need to install Elasticsearch on your EC2 instance. As long as the ElastAlert 2 process can open a socket to the OpenSearch instance, wherever it's hosted, that's what matters.

When you run it in EC2 via Docker, what is the behavior? Does it still immediately exit without any errors?

Have you tried starting from scratch, using the example rules and config files included with ElastAlert 2?

[RLStepper]

Hey @jertel thanks for replying again, I really do appreciate it. I am familiar with Docker and how it works, though I am not sure exactly how Elastalert is ran through Docker (as in, how are rule files specified, are they run upon passing the "docker run" command with the --rule option? Should I enter the container (/bin/bash) and then run elastalert --verbose --config etc etc?)

When I run it via docker, it does indeed exit without any errors, almost immediately. Yes, I have tried starting from scratch using those files, and when I did that I actually tried the Ubuntu AMI on my EC2 instead of the Amazon Linux AMI. Still ran into the same issues described above (ES_HOST = none, etc.)

Throughout this process, I've had various results, but one stood out to me that is a bit confusing. In one of my attempts to run Elastalert, upon running the container, I was prompted to enter my Elasticsearch host, port, SSL T/F, etc. If that information is provided in the config file that is mounted to a volume in the container, why would I need to re-enter it in that scenario? Were those prompts unintentional?

When I entered the relevant info, the result in my CLI looked like I had tried to run every rule in my rules folder, as well as some traceback errors. If you get time, could you provide your terminal history from start to finish of installing and running Elastalert through a docker container? I know that's asking a lot, but maybe I'm missing something or running it incorrectly. The documentation, in my opinion, has a bit of a gap in it as far as that is concerned. It does mention that your elastalert.yaml/config.yaml need to be mounted, as well as mounting a rules folder to /opt/elastalert/rules, but then it just jumps right into technical specifics about the various fields in config. I don't mean that in a negative way, as it's excellent documentation and has helped me tremendously, but there are some underlying assumptions there about the user reading the doc that I am likely missing, so that's on me.

[jertel]

The doc is pretty clear about how to start ElastAlert2 via the container. It's a single command:

As that command shows, both the config file and the rules directory on the host are mounted into the container. As soon as you execute that command the container starts ElastAlert 2 and begins processing all enabled rules in that rules directory. If you see it prompting you for the Elasticsearch host then you are not correctly mounting the config file into the container. You stated you are familiar with Docker so I expect you understand the bind argument fully and I don't need to go into detail on it.

The bottom command docker logs ... is only there to let you see the logs being output to the container stdout.

[RLStepper]

I do understand the bind argument. Thank you for that explanation, that does make it more clear. How would I go about testing a rule using elastalert-test-rule via docker?

[pboushy]

I'm running into the same issue here, but it's intermittent.

I'm in the middle of testing with old data (logs output between 10:00 UTC and 1:00pm UTC on May 31) cause we're moving from ELK 6 to ECK 8, and need to test if ElastAlert2 will work as a replacement for Watchers.

Running elastalert-test-rule rules/db_health_check_timeout-5-in-15m.yaml --days 9 always finds the expected data and says it would generate an alert.

If I then try to run elastalert with the --debug flag, it sometimes finds the data:
elastalert --config /opt/elastalert/config.yaml --start 2023-05-31T10:00:00 --end 2023-05-31T12:59:59 --debug --es_debug_trace=trace.log

If I try to run elastalert with the --verbose flag, it hasn't gotten past "Starting up" once:
elastalert --config /opt/elastalert/config.yaml --start 2023-05-31T10:00:00 --end 2023-05-31T12:59:59 --verbose

Output when it doesn't get past Starting up.

INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
                To send them but remain verbose, use --verbose instead.
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
            To send them but remain verbose, use --verbose instead.
INFO:elastalert:1 rules loaded
INFO:elastalert:Starting up

Details of my setup:
Host: macOS 13.3
Docker Desktop version 4.15.0
Command I run to start the elastalert docker image so I can bounce between elastalert-test-rule command and elastalert easily:

docker run \                                                                              
--mount type=bind,source="$(pwd)"/ElastAlert2/config.yaml,target=/opt/elastalert/config.yaml \
--mount type=bind,source="$(pwd)"/ElastAlert2/smtpauth.yaml,target=/opt/elastalert/smtpauth.yaml \
--mount type=bind,source="$(pwd)"/ElastAlert2/rules/,target=/opt/elastalert/rules \
--mount type=bind,source="$(pwd)"/ElastAlert2/REDACTED.pem,target=/opt/elastalert/REDACTED.pem \
-dit --entrypoint /bin/bash jertel/elastalert2:2.11.0

docker exec -it <containerID> /bin/bash

NOTE - docker run -dit causes it to be interactive initially, and then detach, but I did test by just using -d and then watching logs, AND I've tried having the arguments at the end of the docker command instead of using docker exec to run the commands inside the container.

None of those seem to work.

I'm considering deploying the helm chart to see if this is just a weird Mac + Docker + ElastAlert2 issue that will be resolved by using Helm in k8s, but if you have any ideas, I'd love to be able to test more locally.

Config File:

# This is the folder that contains the rule yaml files
# This can also be a list of directories
# Any .yaml file will be loaded as a rule
rules_folder: /opt/elastalert/rules

# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  minutes: 1

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 15

# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: REDACTED

# The Elasticsearch port
es_port: REDACTED

# Connect with TLS to Elasticsearch
use_ssl: True

# Verify TLS certificates
verify_certs: True

# Show TLS or certificate related warnings
#ssl_show_warn: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See https://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for Elasticsearch
es_username: REDACTED
es_password: REDACTED

# Use SSL authentication with client certificates client_cert must be
# a pem file containing both cert and key for client
ca_certs: /opt/elastalert/REDACTED.pem
#client_cert: /path/to/client_cert.pem
#client_key: /path/to/client_key.key

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

# smtp_host: REDACTED.REDACTED.com
# smtp_port: 587
# smtp_tls: true
# smtp_auth_file: /opt/elastalert/smtpauth.yaml

# Optional timestamp format.
# ElastAlert will print timestamps in alert messages and in log messages using this format.
#custom_pretty_ts_format: '%Y-%m-%d %H:%M'

# Custom logging configuration
# If you want to setup your own logging configuration to log into
# files as well or to Logstash and/or modify log levels, use
# the configuration below and adjust to your needs.
# Note: if you run ElastAlert with --verbose/--debug, the log level of
# the "elastalert" logger is changed to INFO, if not already INFO/DEBUG.
#logging:
#  version: 1
#  incremental: false
#  disable_existing_loggers: false
#  formatters:
#    logline:
#      format: '%(asctime)s %(levelname)+8s %(name)+20s %(message)s'
#
#  handlers:
#    console:
#      class: logging.StreamHandler
#      formatter: logline
#      level: DEBUG
#      stream: ext://sys.stderr
#
#    file:
#      class : logging.FileHandler
#      formatter: logline
#      level: DEBUG
#      filename: elastalert.log
#
#  loggers:
#    elastalert:
#      level: WARN
#      handlers: []
#      propagate: true
#
#    elasticsearch:
#      level: WARN
#      handlers: []
#      propagate: true
#
#    elasticsearch.trace:
#      level: WARN
#      handlers: []
#      propagate: true
#
#    '':  # root logger
#      level: WARN
#      handlers:
#        - console
#        - file
#      propagate: false

rule.yaml

# Alert when the rate of events exceeds a threshold

# (Optional)
# Elasticsearch host
# es_host: elasticsearch.example.com

# (Optional)
# Elasticsearch port
# es_port: 14900

# (OptionaL) Connect with SSL to Elasticsearch
#use_ssl: True

# (Optional) basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword

# (Required)
# Rule name, must be unique
name: Database Health Check Timeout seen 5 times in 15 minutes

# (Required)
# Type of alert.
# the frequency rule type alerts when num_events events occur with timeframe time
type: frequency

# (Required)
# Index to search, wildcard supported
index: REDACTED-REDACTED-*

# (Required, frequency specific)
# Alert when this many documents matching the query occur within a timeframe
num_events: 5

# (Required, frequency specific)
# num_events must occur within this amount of time to trigger an alert
timeframe:
  hours: 1

# (Required)
# A list of Elasticsearch filters used for find events
# These filters are joined with AND and nested in a filtered query
# For more info: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html
filter:
- bool:
    must:
      - term:
          level: ERROR
      - match:
          msg: 
            query: "database health check timed out"
            operator: and

# (Required)
# The alert is use when a match is found
alert:
- "email"

# (required, email specific)
# a list of email addresses to send alerts to
email:
- "[email protected]"

[pboushy]

I don't know why I didn't try this earlier, but if I change:
elastalert --config /opt/elastalert/config.yaml --start 2023-05-31T10:00:00 --end 2023-05-31T12:59:59 --verbose
to:
elastalert --config /opt/elastalert/config.yaml --start 2023-05-31T9:00:00 --verbose

It appears to start working... so elastalert is not running if --end is set (at least in some cases)

[jertel]

According to the client.start() method, the --end param is expected to be in the future. If it's in the past the process will quickly exit. Due to the multi-threaded nature of the scheduler it is possible that a single run could execute before it exits and that perhaps could be explained by the verbose vs debug output causing different delays during processing.

It's possible this was an oversight in the original implementation for historic data processing. Regardless, you will need to omit the --end as you have already discovered.

[pboushy]

Are you ok with me submitting an issue:
"Check if --end value occurs in the past, if so output an error to the user"

[jertel]

I prefer people contribute PRs rather than filing issues. Since this was simple I've already submitted the PR. #1199