It is impossible to setup TLS between Solr and Zookeeper

[jstaf]

Tried setting up solr-operator with TLS between Solr and Zookeeper. (Zookeeper is setup with TLS, and Solr must connect to Zookeeper's TLS port.) It doesn't work or appear to have ever been tested.

Summary of problems I found before giving up:

• Solr cannot connect to Zookeepers using TLSv1.3 (will fail with alert: protocol_version until you downgrade).
• Contrary to the documentation, Solr's initContainers ignore spec.solrZkOpts completely... spec.solrZkOpts sets SOLR_OPTS, but the solr zk and zkCli.sh commands ignore SOLR_OPTS.
  • solr zk uses SOLR_TOOL_OPTS not SOLR_OPTS (and this is specific to Solr 9... Solr 8 uses a different env var) https://github.com/apache/solr/blob/main/solr/bin/solr#L601-L610
  • zkcli.sh uses ZKCLI_JVM_FLAGS not SOLR_OPTS
• The Solr TLS keystores/truststores don't even get mounted into the initContainers (and there is no way to do so), so even if you were able to specify the right combination of flags to solr zk and zkcli.sh in the initContainers, you can't add the certificates to setup TLS to Zookeeper anyways.
• The environment variables for the location of the truststore/keystore/passwords/etc. are not mounted in the initContainer
• There's no documentation on how to setup TLS between Solr and Zookeeper

To reproduce the issue:

Setup Zookeeper with TLS using the Bitnami Helm chart helm install zookeeper bitnami/zookeeper -f zookeeper.yml. Example zookeeper.yml (uses TLS for encryption, but not for authentication):

# note: solr flat-out cannot connect to zookeepers that use TLSv1.3,
# so we downgrade to TLSv1.2 here
jvmFlags: -Dzookeeper.ssl.protocol=TLSv1.2

# disabled for simplicity here
auth:
  client:
    enabled: false
  quorum:
    enabled: false

networkPolicy:
  extraIngress:
    - ports:
        - port: 3181
service:
  # zookeeper healthchecks and stuff are broken without this
  disableBaseClientPort: true
  headless:
    publishNotReadyAddresses: false
tls:
  client:
    # clients encrypt connections with TLS,
    # but do not use mTLS authentication
    enabled: true
    autoGenerated: true
    auth: "none"
  quorum:
    # use full mTLS auth+encryption between servers
    enabled: true
    autoGenerated: true
    auth: "need"

Then install solr-operator, and create a SolrCloud CRD. In this case I've generated some Certificate resources via cert-manger that the SolrCloud is using...

apiVersion: solr.apache.org/v1beta1
kind: SolrCloud
metadata:
  name: solr
spec:
  replicas: 3
  solrImage:
    tag: 9.6.1
  dataStorage:
    persistent:
      reclaimPolicy: Delete
      pvcTemplate:
        spec:
          resources:
            requests:
              storage: "20Gi"
  solrTLS:
    # can't supply just a trustStore on its own, which is annoying
    # (maybe users want to get Solr -> Zookeepr TLS setup first before doing TLS for Solr itself?)
    pkcs12Secret:
      name: solr-tls
      key: keystore.p12
    keyStorePasswordSecret:
      name: solr-tls-password
      key: password
    # doesn't even get mounted into the initcontainer
    trustStoreSecret:
      name: solr-tls
      key: truststore.p12
    trustStorePasswordSecret:
      name: solr-tls-password
      key: password
  solrZkOpts: >
    -Dzookeeper.client.secure=true
    -Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
    -Dzookeeper.ssl.hostnameVerification=false
    -Dzookeeper.ssl.trustStore.location=/var/solr/tls/truststore.p12
    -Dzookeeper.ssl.trustStore.password=some-password
  zookeeperRef:
    connectionInfo:
      externalConnectionString: "zookeeper-0.zookeeper-headless.solr:3181,zookeeper-1.zookeeper-headless.solr:3181,zookeeper-2.zookeeper-headless.solr:3181"
      chroot: /solr

[jstaf]

I got this to work with a custom solr image with a custom version of solr.in.sh and zkcli.sh. Still needs to be fixed in solr-operator though.

solr.in.sh (this change has solr zk auto-trust the zk tls certificate and obey settings from $SOLR_OPTS set by solrZkOpts)

#!/bin/bash
set -eo pipefail

# import zk certificate into default system truststore if not present
if [ ! -f /tmp/cacerts.jks ]; then
    # this step is extremely race-y for some reason
    sleep $((RANDOM % 10))
    echo |\
        openssl s_client -connect $(echo $ZK_SERVER | sed 's/,.*//g') -showcerts |\
        sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/ca.crt
    # we create a copy of the system truststore in /tmp since the root
    # filesystem is not writeable >:(
    cp /opt/java/openjdk/lib/security/cacerts /tmp/cacerts.jks
    keytool -import -noprompt -trustcacerts -alias solr-ca -file /tmp/ca.crt -keystore /tmp/cacerts.jks -storepass changeit
fi

# have "solr zk" and zkcli.sh use $SOLR_OPTS
SOLR_TOOL_OPTS="$SOLR_OPTS -Dzookeeper.ssl.trustStore.location=/tmp/cacerts.jks -Dzookeeper.ssl.trustStore.password=changeit"
ZKCLI_JVM_FLAGS="$SOLR_TOOL_OPTS"

zkcli.sh (make this script actually obey $SOLR_OPTS by sourcing our custom solr.in.sh)

#!/usr/bin/env bash

# You can override pass the following parameters to this script:
#

JVM="java"

# Find location of this script

sdir="`dirname \"$0\"`"

log4j_config="file:$sdir/../../resources/log4j2-console.xml"

solr_home="$sdir/../../solr"

# Settings for ZK ACL
#SOLR_ZK_CREDS_AND_ACLS="-DzkACLProvider=org.apache.solr.common.cloud.DigestZkACLProvider \
#  -DzkCredentialsProvider=org.apache.solr.common.cloud.DigestZkCredentialsProvider \
#  -DzkCredentialsInjector=org.apache.solr.common.cloud.VMParamsZkCredentialsInjector \
#  -DzkDigestUsername=admin-user -DzkDigestPassword=CHANGEME-ADMIN-PASSWORD \
#  -DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=CHANGEME-READONLY-PASSWORD"
# optionally, you can use using a a Java properties file 'zkDigestCredentialsFile'
#...
#   -DzkDigestCredentialsFile=/path/to/zkDigestCredentialsFile.properties
#...

source /etc/default/solr.in.sh  # this is the only change
PATH=$JAVA_HOME/bin:$PATH $JVM $SOLR_ZK_CREDS_AND_ACLS $ZKCLI_JVM_FLAGS -Dlog4j.configurationFile=$log4j_config -Dsolr.home=$solr_home \
-classpath "$sdir/../../solr-webapp/webapp/WEB-INF/lib/*:$sdir/../../lib/ext/*:$sdir/../../lib/*" org.apache.solr.cloud.ZkCLI ${1+"$@"}

[gerlowskija]

Thanks for the detailed writeup @jstaf! I noticed there are some other planned changes in how the operator interacts with ZK here , probably makes sense to tackle this once that's wrapped up?

It seems like you understand the mechanisms here pretty well - is this something you'd be willing to put together a PR for?