feat(openchallenges): add Kibana to the OC stack

[tschaffter]

• Closes [Story] Encrypt communications between ES nodes #2228
• Closes [Story] Add Kibana to the OC stack #2214

Changelog

• Upgrade ES to v8
• Enable ES basic authentication
• Enable ES SSL/TLS encryption
• Add the project openchallenges-kibana

Notes

• ELK projects were initially added in Add elk stack #330
• The user kibana_system is used internally by Kibana. Users are still expected to login in Kibana with their ES credentials.

TODO

• Generate the certificates with the service openchallenges-elasticsearch-setup
• Start the ES nodes with encryption
• Configure Kibana to access ES
• Configure the challenge service to authenticate with ES
• Configure the organization service to authenticate with ES

Resources

• A few tricks on how to set up related Docker images with docker-compose
• Elasticsearch and Kibana installation using Docker Compose
• Elasticsearch & Kibana : security, authentication, and network encryption enabled with Docker Compose
• How to start a Elasticsearch Docker Container with SSL/TLS encryption

Preview

Generate certificates for SSL/TLS encryption

Certificate created by the service openchallenges-elasticsearch-setup:

root@4a4fefe2082a:/certs# ls -alR
.:
total 16
drwxr-x--- 6 root root  169 Oct 12 17:56 .
drwxr-xr-x 1 root root   19 Oct 12 17:56 ..
drwxr-x--- 2 root root   34 Oct 12 17:56 ca
-rw-r----- 1 root root 2510 Oct 12 17:56 ca.zip
-rw-r----- 1 root root 8579 Oct 12 17:56 certs.zip
drwxr-x--- 2 root root   86 Oct 12 17:56 openchallenges-elasticsearch
drwxr-x--- 2 root root  100 Oct 12 17:56 openchallenges-elasticsearch-node-2
drwxr-x--- 2 root root  100 Oct 12 17:56 openchallenges-elasticsearch-node-3

./ca:
total 8
drwxr-x--- 2 root root   34 Oct 12 17:56 .
drwxr-x--- 6 root root  169 Oct 12 17:56 ..
-rw-r----- 1 root root 1200 Oct 12 17:56 ca.crt
-rw-r----- 1 root root 1675 Oct 12 17:56 ca.key

./openchallenges-elasticsearch:
total 8
drwxr-x--- 2 root root   86 Oct 12 17:56 .
drwxr-x--- 6 root root  169 Oct 12 17:56 ..
-rw-r----- 1 root root 1257 Oct 12 17:56 openchallenges-elasticsearch.crt
-rw-r----- 1 root root 1675 Oct 12 17:56 openchallenges-elasticsearch.key

./openchallenges-elasticsearch-node-2:
total 8
drwxr-x--- 2 root root  100 Oct 12 17:56 .
drwxr-x--- 6 root root  169 Oct 12 17:56 ..
-rw-r----- 1 root root 1277 Oct 12 17:56 openchallenges-elasticsearch-node-2.crt
-rw-r----- 1 root root 1675 Oct 12 17:56 openchallenges-elasticsearch-node-2.key

./openchallenges-elasticsearch-node-3:
total 8
drwxr-x--- 2 root root  100 Oct 12 17:56 .
drwxr-x--- 6 root root  169 Oct 12 17:56 ..
-rw-r----- 1 root root 1273 Oct 12 17:56 openchallenges-elasticsearch-node-3.crt
-rw-r----- 1 root root 1679 Oct 12 17:56 openchallenges-elasticsearch-node-3.key

When opening the Discover section

Challenges created over time

Challenge grouped by created_at field

[tschaffter]

Security

When trying ES 8.x, the ES containers fail to start with the following error:

"ERROR", "message":"node validation exception\n[1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.\nbootstrap check failure [1] of [1]: Transport SSL must be enabled if security is enabled. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]"

First, I added the following properties to each ES node:

environment:
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.audit.enabled=true

One of the node then fails with this error:

invalid SSL configuration for xpack.security.transport.ssl - server ssl configuration requires a key and certificate, but these have not been configured; you must set either [xpack.security.transport.ssl.keystore.path], or both [xpack.security.transport.ssl.key] and [xpack.security.transport.ssl.certificate]"

[tschaffter]

Warning message after enabling encryption

There are no error in the ES nodes but this warning message. Probably from the HEALTHCHECK that is done over http.

{"@timestamp":"2023-10-12T18:04:59.070Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/172.29.0.3:9200, remoteAddress=/172.29.0.3:35828}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[openchallenges-elasticsearch-node-2][transport_worker][T#1]","log.logger":"org.elasticsearch.http.netty4.Netty4HttpServerTransport","elasticsearch.cluster.uuid":"bf4DFCUDQlyWS09_i3M5lg","elasticsearch.node.id":"R4Z7CQXgR0ur-Qsd0116dg","elasticsearch.node.name":"openchallenges-elasticsearch-node-2","elasticsearch.cluster.name":"openchallenges-elasticsearch"}

Solution

Use https in the curl command used for the healthcheck. The warning is then no longer shown in the ES nodes.

I can access the health page https://$host:9200/_cat/health?h=status in the browser after accepting the risk related to the self-signed certificate BUT I also need to enter the ES credentials. This may be why the healthcheck still fails.

[tschaffter]

ES node healthcheck error

$ docker inspect --format "{{json .State.Health }}" openchallenges-elasticsearch-node-2 | jq
{
  "Status": "unhealthy",
  "FailingStreak": 47,
  "Log": [
    {
      "Start": "2023-10-12T18:29:37.282351375Z",
      "End": "2023-10-12T18:29:37.420087739Z",
      "ExitCode": 1,
      "Output": "curl: (60) SSL: no alternative certificate subject name matches target host name '192.168.48.2'\nMore details here: https://curl.haxx.se/docs/sslcerts.html\n\ncurl failed to verify the legitimacy of the server and therefore could not\nestablish a secure connection to it. To learn more about this situation and\nhow to fix it, please visit the web page mentioned above.\n"
    },
    {
      "Start": "2023-10-12T18:29:39.432478263Z",
      "End": "2023-10-12T18:29:39.517657068Z",
      "ExitCode": 1,
      "Output": "curl: (60) SSL: no alternative certificate subject name matches target host name '192.168.48.2'\nMore details here: https://curl.haxx.se/docs/sslcerts.html\n\ncurl failed to verify the legitimacy of the server and therefore could not\nestablish a secure connection to it. To learn more about this situation and\nhow to fix it, please visit the web page mentioned above.\n"
    },
    {
      "Start": "2023-10-12T18:29:41.524666287Z",
      "End": "2023-10-12T18:29:41.630094917Z",
      "ExitCode": 1,
      "Output": "curl: (60) SSL: no alternative certificate subject name matches target host name '192.168.48.2'\nMore details here: https://curl.haxx.se/docs/sslcerts.html\n\ncurl failed to verify the legitimacy of the server and therefore could not\nestablish a secure connection to it. To learn more about this situation and\nhow to fix it, please visit the web page mentioned above.\n"
    },
    {
      "Start": "2023-10-12T18:29:43.636282237Z",
      "End": "2023-10-12T18:29:43.760431077Z",
      "ExitCode": 1,
      "Output": "curl: (60) SSL: no alternative certificate subject name matches target host name '192.168.48.2'\nMore details here: https://curl.haxx.se/docs/sslcerts.html\n\ncurl failed to verify the legitimacy of the server and therefore could not\nestablish a secure connection to it. To learn more about this situation and\nhow to fix it, please visit the web page mentioned above.\n"
    },
    {
      "Start": "2023-10-12T18:29:45.768126124Z",
      "End": "2023-10-12T18:29:45.896280954Z",
      "ExitCode": 1,
      "Output": "curl: (60) SSL: no alternative certificate subject name matches target host name '192.168.48.2'\nMore details here: https://curl.haxx.se/docs/sslcerts.html\n\ncurl failed to verify the legitimacy of the server and therefore could not\nestablish a secure connection to it. To learn more about this situation and\nhow to fix it, please visit the web page mentioned above.\n"
    }
  ]
}

[tschaffter]

Using a basic auth token instead of clear username and password:

https://stackoverflow.com/a/45530996

[tschaffter]

Kibana fails to reach ES after enabling SSL encryption for ES

[2023-10-12T21:03:18.730+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. socket hang up - Local: 192.168.112.6:44054, Remote: 192.168.112.5:9200
[2023-10-12T21:03:21.233+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. socket hang up - Local: 192.168.112.6:44176, Remote: 192.168.112.5:9200
[2023-10-12T21:03:23.732+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. socket hang up - Local: 192.168.112.6:53784, Remote: 192.168.112.5:9200

I had a typo in the config of Kibana. Now Kibana loads in the browser but says "Kibana server is not ready yet." but still shows the error above. The UI to login is never shown.

From inside the Kibana container, this command works:

curl -s -u "kibana_system:changeme" --cacert /usr/share/kibana/config/certs/ca/ca.crt https://open
challenges-elasticsearch:9200
{
  "name" : "openchallenges-elasticsearch",
  "cluster_name" : "openchallenges-elasticsearch",
  "cluster_uuid" : "AdD56LOYQ3ysV4FFvEMi7Q",
  "version" : {
    "number" : "8.10.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "6d20dd8ce62365be9b1aca96427de4622e970e9e",
    "build_date" : "2023-09-19T08:16:24.564900370Z",
    "build_snapshot" : false,
    "lucene_version" : "9.7.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

The issue was because Kibana was still trying to access ES over HTTP.

[tschaffter]

The org service now failed to load the data

As expected:

2023-10-13 21:26:47 ERROR [main] o.s.b.w.e.tomcat.TomcatStarter - Error starting Tomcat context. Exception: org.springframework.beans.factory.UnsatisfiedDependencyException. Message: Error creating bean with name 'securityConfiguration': Unsatisfied dependency expressed through method 'setContentNegotationStrategy' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.boot.autoconfigure.web.servlet.WebMvcAutoConfiguration$EnableWebMvcConfiguration': Unsatisfied dependency expressed through method 'setConfigurers' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'openEntityManagerInViewInterceptorConfigurer' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/JpaBaseConfiguration$JpaWebConfiguration.class]: Unsatisfied dependency expressed through method 'openEntityManagerInViewInterceptorConfigurer' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'openEntityManagerInViewInterceptor' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/JpaBaseConfiguration$JpaWebConfiguration.class]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaConfiguration.class]: Invocation of init method failed; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: default] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.search.util.common.SearchException: HSEARCH000520: Hibernate Search encountered failures during bootstrap. Failures:

    default backend: 
        failures: 
          - HSEARCH400080: Unable to detect the Elasticsearch version running on the cluster: HSEARCH400007: Elasticsearch request failed: Connection is closed
            Request: GET  with parameters {}
            Response: (no response)

[tschaffter]

Update 2024-02-22

This PR is almost complete. I believe that one remaining task was how to handle the extra security elements (certs).

I'm closing this PR as we don't have an urgent need to add Kibana to the OC stack.