Unsuccesful startup of ElastAlerts to Elasticsearch v8, due to cert errors

[avatar333]

Good day, all

My thanks to the team who maintains this software, it really does look like a great piece of work. However, I have not been able to get it to work against an Elasticsearch v8x cluster (I spun up a fresh machine with Docker to play around with this) due to issue with find out exactly what certs to use.

Versions in use:

OS:
# cat /etc/redhat-release
Rocky Linux release 8.7 (Green Obsidian)

Elasticsearch:
# rpm -qa | grep elastic
elasticsearch-8.6.2-1.x86_64

Docker:
# docker --version
Docker version 24.0.2, build cb74dfc

Image:
# docker image ls
REPOSITORY                               TAG       IMAGE ID       CREATED       SIZE
ghcr.io/jertel/elastalert2/elastalert2   latest    d5613698125d   4 days ago    411MB

• Configs are in /opt/elastalert2:

# tree /opt/elastalert2/
/opt/elastalert2/
├── certs
│   ├── ca.pem
│   ├── elasticsearch-ca.pem
│   ├── http_ca.crt
│   ├── installation_ca.pem
│   ├── instance.crt
│   └── instance.key
├── elastalert2.sh
├── elastalert.yaml
└── rules.d
    └── bb-poc.yaml

• Config file contents:

# grep -v ^# elastalert.yaml
rules_folder: /opt/elastalert2/rules.d
run_every:
  minutes: 1
buffer_time:
  minutes: 15
es_host: elasticsearch01-stage-hidden
es_port: 9200
use_ssl: True
es_username: elastic
es_password: <password, confirmed as correct>
client_key: /opt/elastalert2/certs/http_ca.crt
writeback_index: elastalert_status
alert_time_limit:
  days: 2

• Docker command:

docker run -d --name elastalert --restart=always \
-v /opt/elastalert2/elastalert.yaml:/opt/elastalert/config.yaml \
-v /opt/elastalert2/rules.d:/opt/elastalert/rules \
ghcr.io/jertel/elastalert2/elastalert2 --verbose

• Firewall ports are confirmed as open between the host running the ElastAlert and the ElasticSearch nodes.

• I installed the cluster 100% according to the installation guide (Using .rpm and followed these steps for the TLS: https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html#rpm-security-configuration [the out-the-box TLS setup method]) and it works fine. I am able to curl from the CLI (of the ElastAlert without issue:

# curl -s --cacert /opt/elastalert2/certs/http_ca.crt -u elastic:$ELASTIC_AUTH "https://elasticsearch01-stage-hidden:9200/_cluster/health?pretty=true"
{
  "cluster_name" : "stage_cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 35,
  "active_shards" : 70,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

• The certs generated by the installation, and which as in use by the nodes, are in /etc/elasticsearch/certs/ (on each elasticsearch host, this is an example of one below):

# ls -al /etc/elasticsearch/certs/
total 28
drwxr-x---. 2 root elasticsearch    62 Feb  8 15:29 .
drwxr-s---. 4 root elasticsearch  4096 Jun  1 11:52 ..
-rw-rw----. 1 root elasticsearch  1915 Feb  8 15:29 http_ca.crt
-rw-rw----. 1 root elasticsearch 10093 Feb  8 15:29 http.p12
-rw-rw----. 1 root elasticsearch  5854 Feb  8 15:29 transport.p12

• I copied the cert that works with the curl and used it for the client_key parameter, without luck

• I tried converting the http.p12 file to a .pem file (from these links), and tried using it for both the ca_certs and client_cert parameter:
  This one: How to use a certificate to connect to an ES cluster #586
  Then from the one above, this one: Rule query is not accurate Yelp/elastalert#3228 (specifically the command openssl pkcs12 -clcerts -nokeys -in http.p12 -out ca.pem)

Docker log error, lots of the same, ending in this:

Traceback (most recent call last):
  File "/usr/local/bin/elastalert-create-index", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elastalert/create_index.py", line 242, in main
    create_index_mappings(es_client=es, ea_index=index, recreate=args.recreate, old_ea_index=old_index)
  File "/usr/local/lib/python3.11/site-packages/elastalert/create_index.py", line 24, in create_index_mappings
    esversion = get_version_from_cluster_info(es_client)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elastalert/util.py", line 567, in get_version_from_cluster_info
    esinfo = client.info()['version']
             ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elasticsearch/client/utils.py", line 152, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elasticsearch/client/__init__.py", line 286, in info
    return self.transport.perform_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elasticsearch/transport.py", line 390, in perform_request
    raise e
  File "/usr/local/lib/python3.11/site-packages/elasticsearch/transport.py", line 358, in perform_request
    status, headers_response, data = connection.perform_request(
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elasticsearch/connection/http_requests.py", line 174, in perform_request
    raise SSLError("N/A", str(e), e)
elasticsearch.exceptions.SSLError: ConnectionError(HTTPSConnectionPool(host='elasticsearch01-stage-hidden', port=9200): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1002)')))) caused by: SSLError(HTTPSConnectionPool(host='elasticsearch01-stage-hidden', port=9200): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1002)'))))
[root@kevinp01-test-bry elastalert2]#

Now, I imagine it's not working due to something stupid I'm doing, but I can't figure out what (certs are not my strong suit).

What files from a default installation method must I use for the ElastAlert config? Or, what additional manual procedure, using the files generated by the default installation, must I complete to convert/transform them for use?

Any assistance would be appreciated!

Regards,
Kevin

[nsano-rururu]

[avatar333]

Good day! Well, as I said, it was something stupid I was doing, as your suggestion did the trick! Thank you very much for the assistance, I look forward to now truly exploring the software!