MirrorMaker2 isn't setting up in AKS cluster

[AAR256]

We're using Strimzi operator version v.027.1 on AKS K8s cluster. When trying to deploy the MirrorMaker2 resource, we don't get any errors but while describing no events are published and it seems to be in frozen state. The YAML file that we're using is below -

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
name: my-mirror-maker-2
spec:
version: 3.0.0
replicas: 1
connectCluster: "west-2"
clusters:

• alias: "north-1"
  bootstrapServers:
• alias: "west-2"
  bootstrapServers:
  config:

  -1 means it will use the default replication factor configured in the broker

  config.storage.replication.factor: -1
  offset.storage.replication.factor: -1
  status.storage.replication.factor: -1
  mirrors:
• sourceCluster: "north-1"
  targetCluster: "west-2"
  sourceConnector:
  config:
  replication.factor: 1
  offset-syncs.topic.replication.factor: 1
  sync.topic.acls.enabled: "false"
  heartbeatConnector:
  config:
  heartbeats.topic.replication.factor: 1
  checkpointConnector:
  config:
  checkpoints.topic.replication.factor: 1
  topicsPattern: "."
  groupsPattern: "."
  resources:
  requests:
  memory: 500Mi
  cpu: "100m"
  limits:
  memory: 600Mi
  cpu: "1"

The logs while describing the kafkamirrormaker2 resource are -

kubectl describe kafkamirrormaker2 my-mirror-maker-2 -n kafka

Resources:
Limits:
Cpu: 1
Memory: 600Mi
Requests:
Cpu: 100m
Memory: 500Mi
Version: 3.0.0
Events:

kubectl get kafkamirrormaker2 -n kafka
NAME DESIRED REPLICAS READY
my-mirror-maker-2 1

[scholzj]

You will need to properly format the YAMLs to make them readable. Without that, nobody can really see how does your configuration look like.

[AAR256]

Thanks for letting me know. I have updated the code. Hope this would make things easy.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: my-mirror-maker-2
spec:
  version: 3.0.0
  replicas: 1
  connectCluster: "west-2"
  clusters:
  - alias: "north-1"
    bootstrapServers: <source-host>
  - alias: "west-2"
    bootstrapServers: <destination-host>
    config:
      # -1 means it will use the default replication factor configured in the broker
      config.storage.replication.factor: -1
      offset.storage.replication.factor: -1
      status.storage.replication.factor: -1
  mirrors:
  - sourceCluster: "north-1"
    targetCluster: "west-2"
    sourceConnector:
      config:
        replication.factor: 1
        offset-syncs.topic.replication.factor: 1
        sync.topic.acls.enabled: "false"
    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 1
    checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 1
    topicsPattern: ".*"
    groupsPattern: ".*"
  resources:
    requests:
      memory: 500Mi
      cpu: "100m"
    limits:
      memory: 600Mi
      cpu: "1"

[scholzj]

I think that config looks good in general ... so you would need to check the logs.

[AAR256]

So, initially we got errors with regards to memory and after increasing that up during the liveliness and readiness probe it's throwing connection refused error and moving to crashloopbackoff state.

  resources:
    requests:
      memory: 8Gi
      cpu: "1"
    limits:
      memory: 10Gi
      cpu: "2"

Kubectl describe pod kafka-mirror-maker-2 -n kafka

Normal   Pulling           4m54s               kubelet             Pulling image "quay.io/strimzi/kafka:0.27.1-kafka-3.0.0"
  Normal   Pulled            4m14s               kubelet             Successfully pulled image "quay.io/strimzi/kafka:0.27.1-kafka-3.0.0" in 40.341453774s
  Warning  Unhealthy         2m5s                kubelet             Liveness probe failed: Get "http://<IP>:8083/": dial tcp <IP>:8083: connect: connection refused
  Warning  Unhealthy         2m5s                kubelet             Readiness probe failed: Get "http://<IP>:8083/": dial tcp <IP>:8083: connect: connection refused
  Warning  BackOff           22s (x5 over 2m2s)  kubelet             Back-off restarting failed container

I have attached the pod logs as well for reference.

2022-03-09 07:29:48,195 WARN The configuration 'config.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,195 WARN The configuration 'producer.security.protocol' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,195 WARN The configuration 'rest.advertised.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,195 WARN The configuration 'group.id' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,195 WARN The configuration 'status.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,195 WARN The configuration 'rest.advertised.port' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,195 WARN The configuration 'client.rack' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,195 WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,195 WARN The configuration 'config.providers' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,195 WARN The configuration 'admin.security.protocol' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,195 WARN The configuration 'header.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,195 WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,195 WARN The configuration 'config.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,195 WARN The configuration 'status.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,196 WARN The configuration 'config.providers.file.class' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,196 WARN The configuration 'offset.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,196 WARN The configuration 'offset.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,196 WARN The configuration 'consumer.security.protocol' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,196 WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,196 WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2022-03-09 07:29:48,196 INFO Kafka version: 3.0.0 (org.apache.kafka.common.utils.AppInfoParser) [main]
2022-03-09 07:29:48,196 INFO Kafka commitId: 8cb0a5e9d3441962 (org.apache.kafka.common.utils.AppInfoParser) [main]
2022-03-09 07:29:48,196 INFO Kafka startTimeMs: 1646810988196 (org.apache.kafka.common.utils.AppInfoParser) [main]
2022-03-09 07:30:18,229 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
2022-03-09 07:30:48,305 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
2022-03-09 07:30:48,306 INFO App info kafka.admin.client for adminclient-1 unregistered (org.apache.kafka.common.utils.AppInfoParser) [kafka-admin-client-thread | adminclient-1]
2022-03-09 07:30:48,306 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: fetchMetadata
2022-03-09 07:30:48,307 INFO [AdminClient clientId=adminclient-1] Timed out 1 remaining operation(s) during close. (org.apache.kafka.clients.admin.KafkaAdminClient) [kafka-admin-client-thread | adminclient-1]
2022-03-09 07:30:48,313 INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2022-03-09 07:30:48,313 INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2022-03-09 07:30:48,313 INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2022-03-09 07:30:48,315 ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed) [main]
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
	at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:70)
	at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:51)
	at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:97)
	at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
	at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
	... 3 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes