Operator crashes without error message after zookeeper is deployed

[kkendzia]

Please use this to only for bug reports. For questions or when you need help, you can use the GitHub Discussions, our #strimzi Slack channel or out user mailing list.

Describe the bug
I set up a new k8s cluster (v1.22.5), deploy the operator 0.27.0 and test deploy the example (persistent or ephemeral).
When the deployment of zookeeper finishes, the operator just crashes without any error message, leading to a CrashLoopBackOff.

To Reproduce
Steps to reproduce the behavior:

1. Deploy operator 0.27.0 (also 0.26.1 crashed) on on-prem k8s 1.22.5
2. Deploy example
3. wait for zookeeper to be deployed

Expected behavior
No crash and a successful deployment.

Environment (please complete the following information):

• Strimzi version: 0.27.0
• Installation method: helm
• Kubernetes cluster: k8s 1.22.5
• Infrastructure: On-premise, 3 master, 3 worker

YAML files and logs

2022-01-13 14:09:15 DEBUG ZookeeperScaler:76 - Reconciliation #1(watch) Kafka(kafka-test/my-cluster): Creating Zookeeper Scaler for cluster my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka-test.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka-test.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka-test.svc:2181
2022-01-13 14:09:15 DEBUG Util:118 - Reconciliation #1(watch) Kafka(kafka-test/my-cluster): Waiting for ZooKeeperAdmin connection to my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka-test.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka-test.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka-test.svc:2181 to get connected
2022-01-13 14:09:15 TRACE Util:129 - Reconciliation #1(watch) Kafka(kafka-test/my-cluster): ZooKeeperAdmin connection to my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka-test.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka-test.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka-test.svc:2181 is not connected

After this message the operator crashes

Additional context
Seems like an OOM issue, because after increasing the resources to 1Gi memory it works out without problems.
Maybe the default configuration should increase the memory.

[scholzj]

The reason why the pod was killed should be in the events. So you should be able to check it there to confirm.

[kkendzia]

Sorry this was my fault. Multiple replicasets due to multiple values configs I tested. Please ignore the deployment events.

Deployment events:

Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  4m21s  deployment-controller  Scaled up replica set strimzi-cluster-operator-745cdbb6fd to 1

Pod events:

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  4m28s                 default-scheduler  Successfully assigned kafka-operator/strimzi-cluster-operator-745cdbb6fd-psx2n to <host>
  Normal   Pulled     104s (x5 over 4m27s)  kubelet            Container image "quay.io/strimzi/operator:0.27.0" already present on machine
  Normal   Created    104s (x5 over 4m27s)  kubelet            Created container strimzi-cluster-operator
  Normal   Started    103s (x5 over 4m27s)  kubelet            Started container strimzi-cluster-operator
  Warning  BackOff    94s (x10 over 3m33s)  kubelet            Back-off restarting failed container

This was a fresh install of operator and pod.

[scholzj]

So why does it fail? What is in the logs? The pod events do not suggest anything about being out of resources.

[kkendzia]

As written in the original message, the operator just crashes, without any logs. Only this:

2022-01-13 14:09:15 DEBUG ZookeeperScaler:76 - Reconciliation #1(watch) Kafka(kafka-test/my-cluster): Creating Zookeeper Scaler for cluster my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka-test.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka-test.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka-test.svc:2181
2022-01-13 14:09:15 DEBUG Util:118 - Reconciliation #1(watch) Kafka(kafka-test/my-cluster): Waiting for ZooKeeperAdmin connection to my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka-test.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka-test.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka-test.svc:2181 to get connected
2022-01-13 14:09:15 TRACE Util:129 - Reconciliation #1(watch) Kafka(kafka-test/my-cluster): ZooKeeperAdmin connection to my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka-test.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka-test.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka-test.svc:2181 is not connected

And then it kicks me out of kubectl logs - this is generated with loglevel TRACE
Being out of resources was just an assumption of mine due to "fixing" this behaviour when increasing the memory to 1Gi

[scholzj]

Ok. I don't know - never saw this before I'm afraid. OoM normally records in the events. So it does when the healthchecks are not responding.

[kkendzia]

That's what is my expectation, too.