Connection issue of elastalert2 with elasticsearch

[Nabeel-Akhtar-Qbatch]

I am trying to connect elastalert2 with elasticsearch but unable to make connection

below is my docker-compose file

version: '3.6'
services:
Elasticsearch:
image: elasticsearch:7.17.0
container_name: elasticsearch
restart: always
volumes:
- elastic_data:/usr/share/elasticsearch/data/
environment:
ES_JAVA_OPTS: "-Xmx256m -Xms256m"
discovery.type: single-node
xpack.security.enabled: "true"
xpack.security.authc.api_key.enabled: "true"
ports:
- '9200:9200'
- '9300:9300'
networks:
- elk

Logstash:
image: logstash:7.17.0
container_name: logstash
restart: always
volumes:
- ./logstash/:/logstash/
command: logstash -f /logstash/logstash.conf
ports:
- '5044:5044'
- '9600:9600'
environment:
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
networks:
- elk

Kibana:
image: kibana:7.17.0
container_name: kibana
restart: always
ports:
- '5601:5601'
environment:
- ELASTICSEARCH_URL=http://elasticsearch:9200
- ELASTICSEARCH_USERNAME=elastic
- ELASTICSEARCH_PASSWORD=***
depends_on:
- Elasticsearch
networks:
- elk

elastalert:
image: jertel/elastalert2
container_name: elastalert2
volumes:
- ./elastalert/config.yaml:/opt/elastalert/config.yaml
- ./elastalert/rules:/opt/elastalert/rules
# - ./elastalert/data:/opt/elastalert/data
environment:
- ELASTALERT_CONFIG=/opt/elastalert/config.yaml
- SET_CONTAINER_TIMEZONE=true
- CONTAINER_TIMEZONE=UTC
depends_on:
- Elasticsearch
networks:
- elk

volumes:
elastic_data: {}

networks:
elk

########################
Below is my config.yaml

This is the folder that contains the rule yaml files

rules_folder: /opt/elastalert/rules

run_every:
minutes: 5

buffer_time:
minutes: 15

Connect with TLS to Elasticsearch

use_ssl: False

Verify TLS certificates

verify_certs: False
es_host: Elasticsearch
es_port: 9200
es_username: elastic
es_password: ***
writeback_index: elastalert_status

alert_time_limit:
days: 2

##################################################
I am unable to make connection of elastalert2 with elasticsearch.

Getting below error in elastalert2 logs
NING:elasticsearch:GET http://Elasticsearch:9200/ [status:N/A request:0.021s]
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/usr/local/lib/python3.12/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

elasticsearch.exceptions.ConnectionError: ConnectionError(HTTPConnectionPool(host='elasticsearch', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd8567d1f70>: Failed to establish a new connection: [Errno 111] Connection refused'))) caused by: ConnectionError(HTTPConnectionPool(host='elasticsearch', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd8567d1f70>: Failed to establish a new connection: [Errno 111] Connection refused')))

[Nabeel-Akhtar-Qbatch]

@nsano-rururu please help me out with it

[nsano-rururu]

Since I have no knowledge of k8s, I will only ask basic questions.

In the es_host below, Elasticsearch with a capital letter at the beginning is specified,
kibana's ELASTICSEARCH_URL=http://elasticsearch:9200 is specified in all lowercase letters.
Are you able to log in and connect to elasticsearch with kibana without any problems?
Also, if I change the values of the settings below to elasticsearch in all lowercase letters and deploy again, will the connection error still occur?

es_host: Elasticsearch

[Nabeel-Akhtar-Qbatch]

ELK is running perfectly. There is no issue with it.

Are you able to log in and connect to elasticsearch with kibana"
Yes, it is working fine

Also, if I change the values of the settings below to elasticsearch in all lowercase letters and deploy again, will the connection error still occur?
I already tried this but it did not work

[nsano-rururu]

I have no knowledge of k8s, so I can't pinpoint the cause.
I am terribly sorry.

@jertel

Do you know the cause?

[Nabeel-Akhtar-Qbatch]

Actually I am not using k8's , I am using docker container.
ELK all containers are running but elastalert is not making connection with elasticsearch.

[nsano-rururu]

The sample below does not support password settings for elasticsearch, so some customization is required.

cd /home/sano2/dkwork2/es/es
mkdir -p ./mariadb/data
chmod -R 777 ./mariadb/data
mkdir -p ./mariadb/log
chmod -R 777 ./mariadb/log
mkdir -p rules rule_templates
chmod -R 777 rules rule_templates

/home/sano2/dkwork2/es/es
|--Dockerfiles
|  |--Dockerfile.elastalert
|--docker-compose.yml
|--es
|  |--config
|  |  |--elasticsearch.yml
|  |--data
|--fluentd
|  |--dockerfiles
|  |  |--Dockerfile
|  |--etc
|  |  |--fluent.conf
|--kibana
|  |--config
|  |  |--kibana.yml
|--mariadb
|  |--etc
|  |  |--mymariadb.cnf
|  |--log
|  |  |--error.log
|  |  |--general.log
|  |  |--slow.log
|--work
|  |--bin
|  |  |--elastalert-start.sh
|  |  |--elastic_search_status.sh
|  |--config
|  |  |--elastalert.yaml
|  |--rule_templates
|  |--rules
|  |  |--a.yaml

docker-compose.yml

version: "3.7"
services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
    restart: always
    volumes:
      #- ./es/data:/usr/share/elasticsearch/data
      - ./es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:9200 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 180s

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:8.11.0
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s
 
  fluentd:
    container_name: fluentd
    image: fluentd:1.15.3
    build: ./fluentd/dockerfiles
    ports:
      - 24224:24224
      - 24224:24224/udp
    environment:
      - FLUENTD_CONF=fluent.conf
    volumes:
      - ./fluentd/etc/fluent.conf:/fluentd/etc/fluent.conf
      - ./mariadb/log:/var/log/mysql
    user: root
    restart: unless-stopped

  mariadb:
    container_name: mariadb
    image: mariadb:10.4.11
    ports:
      - 3306:3306
    environment:
      - MYSQL_ROOT_PASSWORD=mariadb
    volumes:
      - ./mariadb/etc:/etc/mysql/conf.d
      - ./mariadb/log:/var/log/mysql
    restart: unless-stopped
    user: root
    healthcheck:
        test: ["CMD-SHELL", "mysqladmin -h 'localhost' -u root -pmariadb ping --silent"]
        interval: 30s
        timeout: 30s
        retries: 3

  elastalert:
    container_name: elastalert
    build:
      context: .
      dockerfile: Dockerfiles/Dockerfile.elastalert
    image: elastalert:2.15.0
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./work/config/elastalert.yaml:/opt/elastalert/config.yaml
      - ./work/rules:/opt/elastalert/rules
      - ./work/rule_templates:/opt/elastalert/rule_templates

Dockerfile.elastalert

FROM jertel/elastalert2:2.15.0
USER root

RUN apt-get update && \
    apt-get install -y vim bash curl && \
    rm -rf /var/lib/apt/lists/*

ADD praeco/bin/elastalert-start.sh /usr/local/bin/
ADD praeco/bin/elastic_search_status.sh /usr/local/bin/

RUN chmod +x /usr/local/bin/elastalert-start.sh 
RUN chmod +x /usr/local/bin/elastic_search_status.sh

USER elastalert
ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]

elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0

kibana.yml

server.name: kibana
server.host: "0.0.0.0"
server.shutdownTimeout: "5s"
elasticsearch.hosts: http://elasticsearch:9200
monitoring.ui.container.elasticsearch.enabled: true

(fluend) Dockerfile

FROM fluent/fluentd:v1.15.3-debian-1.0

# Use root account to use apt
USER root

# below RUN includes plugin as examples elasticsearch is not required
# you may customize including plugins as you wish
RUN buildDeps="sudo make gcc g++ libc-dev" \
 && apt-get update \
 && apt-get install -y --no-install-recommends $buildDeps \
 && sudo gem install fluent-plugin-elasticsearch -v 5.2.4 \
 && sudo gem install fluent-plugin-mysqlslowquery -v 0.0.9 \
 && sudo gem install elasticsearch -v 8.6.0 \
 && sudo gem install elasticsearch-api -v 8.6.0 \
 && sudo gem sources --clear-all \
 && SUDO_FORCE_REMOVE=yes \
    apt-get purge -y --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

USER fluent

fluent.conf

<source>
  @type mysql_slow_query
  path /var/log/mysql/slow.log
  pos_file /tmp/mysql/slow.pos
  tag mysqld.slow_query
   <parse>
     @type none
   </parse>
</source>
<source>
  @type tail
  format none
  path /var/log/mysql/general.log
  pos_file /tmp/mysql/general.pos
  tag mysqld.general
</source>
<source>
  @type tail
  format none
  path /var/log/mysql/error.log
  pos_file /tmp/mysql/error.pos
  tag mysqld.error
</source>

<match **.**>
  @type copy
  <store>
    @type stdout
  </store>
  <store>
    @type elasticsearch
    include_tag_key true
    tag_key @log_name
    logstash_format true
    logstash_prefix mariadb-log
    host elasticsearch
    port 9200
    index_name mysql
    flush_interval 10s
  </store>
</match>

mymariadb.cnf

[mysqld]
general_log
general_log_file=/var/log/mysql/general.log
slow_query_log
slow_query_log_file=/var/log/mysql/slow.log
long_query_time=5
log-queries-not-using-indexes
log-error=/var/log/mysql/error.log

elastalert-start.sh

#!/bin/bash

set -e

echo "Giving Elasticsearch at $ELASTICSEARCH_URL time to start..."

elastic_search_status.sh

echo "Starting ElastAlert2!"

bash /opt/elastalert/run.sh

elastic_search_status.sh

#!/bin/bash

set -e

if [ $# -gt 0 ]; then
  ES_URL="$1"
elif [[ -n $ELASTICSEARCH_URL ]]; then
  ES_URL="$ELASTICSEARCH_URL"
elif [[ -n $ES_HOST ]] && [[ -n $ES_PORT ]]; then
  ES_URL="http://$ES_HOST:$ES_PORT"
else
  ES_URL="http://elasticsearch:9200"
fi

until [[ "$(curl -fsSL "$ES_URL/_cat/health?h=status" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')" =~ ^(yellow|green)$ ]]; do
  # printf '+' >&2
  sleep 1
done

echo "Elasticsearch is up and healthy at "$ES_URL"" >&2

elastalert.yaml

es_host: elasticsearch
es_port: 9200
rules_folder: rules
run_every:
  seconds: 60
buffer_time:
  minutes: 1
writeback_index: elastalert_status
alert_time_limit:
  days: 2
skip_invalid: True

a.yaml

alert:
  - "slack"
alert_subject: "Test {0}\_\n\n123\n\_b{1}\_aa\n\nfoo\n\nbar\n\n--baz"
alert_text: "Test {0}\_\n\n123\n\_b{1}\_aa\n\nfoo\n\nbar\n\n--baz"
filter:
  - query:
      query_string:
        query: "@timestamp:*"
slack_webhook_url: 'https://hooks.slack.com/services/xxxxxxxxxx'
index: "mariadb-log-*"
is_enabled: true
name: "a"
num_events: 2
realert:
  minutes: 5
slack_channel_override:
  - "#abc"
slack_emoji_override: ":kissing_cat:"
slack_msg_color: "warning"
slack_parse_override: "none"
slack_title_link: "http://127.0.0.1:8080/rules/a"
slack_username_override: "elastalert"
terms_size: 50
timeframe:
  minutes: 5
timestamp_field: "@timestamp"
timestamp_type: "iso"
type: "frequency"
use_strftime_index: false

[Nabeel-Akhtar-Qbatch]

@jertel please help me out with it

[jertel]

Perhaps when you bring up the full 4 container cluster the ES server isn't fully initialized, so ElastAlert2 can't connect and exits.

Have you tried adding restart: always to the ElastAlert2 docker compose block? Or have you tried manually restarting the ElastAlert2 container a few minutes after ES starts?