AMQ Broker Operator Helm Enhancement

The aim is to automate the creation of the AMQ Broker Custom Resources and at the same time enhance the broker.xml with a configuration that is not yet available in the operator. The helm chart keeps in sync the custom resources and the custom broker.xml.

Prerequisites

1. AMQ Broker Operator must running in your target namespace.

Install AMQ Broker Helm Charts

helm upgrade --install mycluster amq-broker/

Test it

helm test mycluster

Custom Certificate Authority

The helm chart provides out of the box self signed certificate for each acceptor defined, but in case you want provide your custom certificate you need to create in advance the secret.

mkdir -p tls/{ca,broker}

Certificate Autority

This section is to simulate a private certificate authority.

1. Certificate Authority (CA)

   openssl req -new -newkey rsa:2048 -x509 -keyout tls/ca/ca.key -out tls/ca/ca.crt -days 365 -subj "/CN=mycompany.com"

2. Create a Truststore

       keytool -import -storepass password -file tls/ca/ca.crt -alias mycompany.com -keystore tls/ca/client.ts

Broker Certificate

1. Broker Key

   keytool -genkeypair -storepass password -keyalg RSA -keysize 2048 -dname "CN=mycluster-amq-broker-*-svc-rte-brk.apps-crc.testing" -ext san="dns:mycluster-amq-broker-all-0-svc-rte-brk.apps-crc.testing,dns:mycluster-amq-broker-all-0-svc.brk.svc" -alias broker -keystore tls/broker/broker.ks

2. Certificate Signing Request (CSR)

   keytool -certreq -storepass password -keyalg rsa -alias broker -keystore tls/broker/broker.ks -file tls/broker/tls.csr

3. Certificate Authority Sign

   openssl x509 -req -CA tls/ca/ca.crt -CAkey tls/ca/ca.key -in tls/broker/tls.csr -out tls/broker/tls.crt -days 365 -CAcreateserial
   
   keytool -import -v -trustcacerts -alias root -file tls/ca/ca.crt -keystore tls/broker/broker.ks
   keytool -import -v -trustcacerts -alias broker -file tls/broker/tls.crt -keystore tls/broker/broker.ks

   1. Verify the chain

      keytool -list -v -keystore tls/broker/broker.ks

4. Import to truststore

   keytool -import -storepass password -file tls/broker/tls.crt -alias server -keystore tls/ca/client.ts

Create Secret

Note	Naming convention: <CustomResourceName>-<AcceptorName>-secret

oc create secret generic mycluster-amq-broker-all-secret \
  --from-file=tls/broker \
  --from-file=tls/ca/client.ts \
  --from-file=tls/ca/ca.crt \
  --from-literal=keyStorePassword=password \
  --from-literal=trustStorePassword=password

Important

Adopt with Helm

oc annotate secret/mycluster-amq-broker-all-secret meta.helm.sh/release-name=mycluster --overwrite
oc annotate secret/mycluster-amq-broker-all-secret meta.helm.sh/release-namespace=brk --overwrite
oc label secret/mycluster-amq-broker-all-secret app.kubernetes.io/managed-by=Helm --overwrite

Install with Custom Certificate

helm upgrade --install mycluster --set pki.self=false

Test External Client Communication

1. Extract Client TrustStore

   oc extract secret/mycluster-amq-broker-all-secret --keys=client.ts

2. Extract AMQ Endpoint

   export AMQ_ENDPOINT=$(oc get route mycluster-amq-broker-all-0-svc-rte --no-headers -o custom-columns=:.spec.host)

3. Launch Artemis Producer Command

   ./artemis producer --user amq --password amq --message-count 10 --url="tcp://${AMQ_ENDPOINT}:443?sslEnabled=true;trustStorePath=./client.ts;trustStorePassword=password"
   
   Connection brokerURL = tcp://mycluster-amq-broker-all-0-svc-rte-brk.apps-crc.testing:443?sslEnabled=true;trustStorePath=./client.ts;trustStorePassword=password
   Producer ActiveMQQueue[TEST], thread=0 Started to calculate elapsed time ...
   
   Producer ActiveMQQueue[TEST], thread=0 Produced: 10 messages
   Producer ActiveMQQueue[TEST], thread=0 Elapsed time in second : 0 s
   Producer ActiveMQQueue[TEST], thread=0 Elapsed time in milli second : 79 milli seconds

4. Launch Artemis Consumer Command

./artemis consumer --user amq --password amq --message-count 10 --url="tcp://${AMQ_ENDPOINT}:443?sslEnabled=true;trustStorePath=./client.ts;trustStorePassword=password"

Connection brokerURL = tcp://mycluster-amq-broker-all-0-svc-rte-brk.apps-crc.testing:443?sslEnabled=true;trustStorePath=./client.ts;trustStorePassword=password
Consumer:: filter = null
Consumer ActiveMQQueue[TEST], thread=0 wait until 10 messages are consumed
Consumer ActiveMQQueue[TEST], thread=0 Consumed: 10 messages
Consumer ActiveMQQueue[TEST], thread=0 Elapsed time in second : 0 s
Consumer ActiveMQQueue[TEST], thread=0 Elapsed time in milli second : 17 milli seconds
Consumer ActiveMQQueue[TEST], thread=0 Consumed: 10 messages
Consumer ActiveMQQueue[TEST], thread=0 Consumer thread finished

Bridge Scenario

1. Create two OpenShift project.

   oc new-project brk1
   oc new-project brk2

2. Install AMQ Broker Operator for both the projects.

3. Create a NetworkPolicy to allow connection from brk1 to brk2.

   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
     name: mycluster1-amq-broker-all
     labels:
       app.kubernetes.io/name: amq-broker
       app.kubernetes.io/instance: mycluster2
   spec:
     podSelector:
       matchLabels:
         app.kubernetes.io/name: amq-broker
         app.kubernetes.io/instance: mycluster2
     ingress:
       - from:
           - namespaceSelector:
               matchLabels:
                 app.kubernetes.io/instance: mycluster
         ports:
         - port: 61617
           protocol: TCP

4. Label brk1 namespace with app.kubernetes.io/instance: mycluster

   oc label namespace brk1 app.kubernetes.io/instance=mycluster

5. Install AMQ Broker Helm Chart on brk2

   helm upgrade --install mycluster2 amq-broker -n brk2 -f examples/bridge/values.brk2.yaml

6. Test the installation on brk2

   helm test mycluster2 -n brk2

7. In case of SSL Connector to brk2, create a secret on brk1 to able to connect to brk2.

   mkdir -p examples/bridge/brk2-certs
   
   oc extract secret/mycluster2-amq-broker-all-secret --to=examples/bridge/brk2-certs -n brk2
   
   oc create secret generic mycluster2-amq-broker-all-brk2-secret --from-file=examples/bridge/brk2-certs/ -n brk1

8. Install AMQ Broker Helm Chart on brk1

   helm upgrade --install mycluster amq-broker -n brk1 -f examples/bridge/values.brk1.yaml

9. Test the installation on brk1

   helm test mycluster -n brk1

Test the bridge connection

Note	Check Test External Client Communication to launch producer and consumer command.

1. Produce message on brk1

./artemis producer --user amq --password amq --message-count 10 --url="tcp://${AMQ_ENDPOINT}:443?sslEnabled=true;trustStorePath=./client.ts;trustStorePassword=password" --destination=bridge.test.address

1. Consume message on brk2

./artemis consumer --user amq --password amq --message-count 10 --url="tcp://${AMQ_ENDPOINT}:443?sslEnabled=true;trustStorePath=./client.ts;trustStorePassword=password" --destination=bridge.test.forwarding

Uninstall it

helm uninstall mycluster --no-hooks

Failed Status

In case of failed installation due to missing role rights, AMQ Broker operator, tests failed or others, please make sure to clean up all the pending resources.

oc delete all -lapp.kubernetes.io/name=amq-broker

Progress

Table 1. Progress

Task	Status
Add Address CRD logic	DONE
Patch AMQ Broker with custom dynamic configuration	DONE
Test Internal Connection Implementation	DONE
SSL Selfsigned Implementation	DONE
Test with External Client Implementation	DONE
Keystore and Truststore Password	DONE
SSL Custom CA Implementation	DONE #2
User Management	DONE #3
Diverts Implementation	DONE #1
Bridge Implementation	#1
Network Policy	#4
Make sure that the operator creates all the Addresses	TODO
High Availability and How scale down controller actives without the operator.	TODO
E2E Test- Migrate one standalone broker	TODO

Pseudo Code

1. Install Operator to specific namespace on the cluster

   1. This can be provided by the cluster-admin as namespaced installation and should give the right role to manage the AMQ Broker CRD.

      Note	You cannot create more than one broker deployment in a given OpenShift project by deploying multiple broker Custom Resource (CR) instances. However, when you have created a broker deployment in a project, you can deploy multiple CR instances for addresses. Reference.

2. [pre-install hook] Restore AMQ Broker Operator

3. [install/upgrade] Create kind: ActiveMQArtemis and ActiveMQArtemisAddress

4. [install/upgrade] Create Config Map with custom broker xml.

5. [post-install hook] Shutdown the AMQ Broker Operator

6. [post-install hook] Adopt the AMQ Broker resource to Helm:

   annotations:
     meta.helm.sh/release-name: release-name
     meta.helm.sh/release-namespace: namespace-name
   labels:
     app.kubernetes.io/managed-by: Helm

7. [post install] Adjust AMQ Broker Stateful set to use the custom broker xml.

   1. Set BROKER_XML environment variable with your custom broker.xml.

8. [test] Verify the installation is correct.

Note	A *-hook install image requires oc client quay.io/openshift/origin-cli:4.6 and running with edit role on the specific namespace.

Important

• In AMQ Broker 7.7, if you want to configure any of the following items, you must add the appropriate configuration to the main CR instance before deploying the CR for the first time.

  • Address settings

  • The size of the Persistent Volume Claim (PVC) required by each broker in a deployment for persistent storage

  • Limits and requests for memory and CPU for each broker in a deployment

• During an active scaling event, any further changes that you apply are queued by the Operator and executed only when scaling is complete. For example, suppose that you scale the size of your deployment down from four brokers to one. Then, while scaledown is taking place, you also change the values of the broker administrator user name and password. In this case, the Operator queues the user name and password changes until the deployment is running with one active broker.

• All CR changes – apart from changing the size of your deployment, or changing the value of the expose attribute for acceptors, connectors, or the console – cause existing brokers to be restarted. If you have multiple brokers in your deployment, only one broker restarts at a time.

• To configure address and queue settings for broker deployments on OpenShift Container Platform, you add configuration to an addressSettings section of the main Custom Resource (CR) instance for the broker deployment. This contrasts with standalone deployments on Linux or Windows, for which you add configuration to an address-settings element in the broker.xml configuration file.

• The format used for the names of configuration items differs between OpenShift Container Platform and standalone broker deployments. For OpenShift Container Platform deployments, configuration item names are in camel case, for example, defaultQueueRoutingType. By contrast, configuration item names for standalone deployments are in lower case and use a dash (-) separator, for example, default-queue-routing-type.

• Addresses are created by the AMQ Broker Operator using Artemis Jolokia and MBean.

Reference

• https://access.redhat.com/documentation/en-us/red_hat_amq/7.7/html-single/deploying_amq_broker_on_openshift/index#con-br-configuring-broker-certificate-for-hostname-verification_broker-ocp