Operator restarts frequently

[JonathanGildevall]

Hi,

I have an issue where the operator restarts frequently, causing "Back-off restarting failed container" errors. The EMQX instance seems unaffected.

See the describe output of the pod:

Namespace:        cloud-communicator
Priority:         0
Service Account:  emqx-operator
Node:             gke-primary-emqx-operator-6f751ad5-3vdp/10.132.0.34
Start Time:       Tue, 17 Jan 2023 14:54:29 +0100
Labels:           app.kubernetes.io/instance=emqx-operator
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=emqx-operator
                  app.kubernetes.io/version=2.1.0
                  control-plane=controller-manager
                  helm.sh/chart=emqx-operator-2.1.0
                  pod-template-hash=5cdb6dc8d9
Annotations:      <none>
Status:           Running
IP:               10.88.4.10
IPs:
  IP:           10.88.4.10
Controlled By:  ReplicaSet/emqx-operator-controller-manager-5cdb6dc8d9
Containers:
  manager:
    Container ID:  containerd://17c0084d55e2def06caa1ab377e335bb6b6fef4e3ce383efc972ede3d03e0461
    Image:         emqx/emqx-operator-controller:2.1.0
    Image ID:      docker.io/emqx/emqx-operator-controller@sha256:d2846b145f3fa276cbee1253f7a0ac547288bee1c999699eaaaaa1b949cbd7e0
    Ports:         8080/TCP, 9443/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /manager
    Args:
      --leader-elect
      --metrics-bind-address=:8080
      --health-probe-bind-address=:8081
    State:          Running
      Started:      Mon, 23 Jan 2023 10:17:28 +0100
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Sun, 22 Jan 2023 23:09:04 +0100
      Finished:     Mon, 23 Jan 2023 10:17:23 +0100
    Ready:          True
    Restart Count:  28
    Limits:
      cpu:     100m
      memory:  100Mi
    Requests:
      cpu:        100m
      memory:     30Mi
    Liveness:     http-get http://:8081/healthz delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:    http-get http://:8081/readyz delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /tmp/k8s-webhook-server/serving-certs from cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zt76c (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  emqx-operator-webhook-server-cert
    Optional:    false
  kube-api-access-zt76c:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              cloud.google.com/gke-nodepool=emqx-operator
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From     Message
  ----     ------     ----                   ----     -------
  Warning  Unhealthy  29m (x119 over 5d11h)  kubelet  Liveness probe failed: Get "http://10.88.4.10:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Normal   Killing    29m (x16 over 5d10h)   kubelet  Container manager failed liveness probe, will be restarted
  Warning  Unhealthy  29m (x368 over 5d11h)  kubelet  Readiness probe failed: Get "http://10.88.4.10:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Also, the cert manager seems to restart as well, but not as often:

cert-manager-55dd688c-6jl6f                         1/1     Running   3 (2d1h ago)    8d
cert-manager-cainjector-7d9466748-p7nck             1/1     Running   15 (2d1h ago)   8d
cert-manager-webhook-d77bbf4cb-sqgjx                1/1     Running   0               8d
emqx-core-0                                         1/1     Running   0               5d19h
emqx-operator-controller-manager-5cdb6dc8d9-lbf5p   1/1     Running   28 (34m ago)    5d19h

I'm running the latest 2.1 version installed without any modifications on a GKE cluster

[Rory-Z]

Hi @JonathanGildevall Could you please show the lastState of EMQX Operator container, you can running kubectl get pods $PodName -n $NameSpace -o json | jq ".status.containerStatuses"

[JonathanGildevall]

Thanks for the quick reply!

[
  {
    "containerID": "containerd://8ec480e2e9d391253583d907f8128989f6c32ddb8aef93689a857ca152d48f23",
    "image": "docker.io/emqx/emqx-operator-controller:2.1.0",
    "imageID": "docker.io/emqx/emqx-operator-controller@sha256:d2846b145f3fa276cbee1253f7a0ac547288bee1c999699eaaaaa1b949cbd7e0",
    "lastState": {
      "terminated": {
        "containerID": "containerd://b6cac3a927b97dfeea8f5ab84fab3e3c03a7f47c83d820bb5dc298c157f64224",
        "exitCode": 137,
        "finishedAt": "2023-01-24T04:49:42Z",
        "reason": "Error",
        "startedAt": "2023-01-23T19:25:44Z"
      }
    },
    "name": "manager",
    "ready": true,
    "restartCount": 30,
    "started": true,
    "state": {
      "running": {
        "startedAt": "2023-01-24T04:49:43Z"
      }
    }
  }
]

Seeing as this is an OOM error, guess the solution is to increase the resources? Even though the nodepool should have sufficient cpu/memory to cope with the operator + cert-manager

[JonathanGildevall]

Switched to a bigger instance, but still experience restarts:

[
  {
    "containerID": "containerd://4672622ebc7678f21b8a86bf1c01eeb0557fa9daf80532630eaa4d18436ef4ff",
    "image": "docker.io/emqx/emqx-operator-controller:2.1.0",
    "imageID": "docker.io/emqx/emqx-operator-controller@sha256:d2846b145f3fa276cbee1253f7a0ac547288bee1c999699eaaaaa1b949cbd7e0",
    "lastState": {
      "terminated": {
        "containerID": "containerd://708bfed04a74479dc03b2a2650c8d72a190bd3c747ea6ed232263237a1b448a0",
        "exitCode": 137,
        "finishedAt": "2023-01-31T05:05:25Z",
        "reason": "OOMKilled",
        "startedAt": "2023-01-30T19:54:55Z"
      }
    },
    "name": "manager",
    "ready": true,
    "restartCount": 2,
    "started": true,
    "state": {
      "running": {
        "startedAt": "2023-01-31T05:05:26Z"
      }
    }
  }
]

[Gala-R]

Switched to a bigger instance, but still experience restarts:

[
  {
    "containerID": "containerd://4672622ebc7678f21b8a86bf1c01eeb0557fa9daf80532630eaa4d18436ef4ff",
    "image": "docker.io/emqx/emqx-operator-controller:2.1.0",
    "imageID": "docker.io/emqx/emqx-operator-controller@sha256:d2846b145f3fa276cbee1253f7a0ac547288bee1c999699eaaaaa1b949cbd7e0",
    "lastState": {
      "terminated": {
        "containerID": "containerd://708bfed04a74479dc03b2a2650c8d72a190bd3c747ea6ed232263237a1b448a0",
        "exitCode": 137,
        "finishedAt": "2023-01-31T05:05:25Z",
        "reason": "OOMKilled",
        "startedAt": "2023-01-30T19:54:55Z"
      }
    },
    "name": "manager",
    "ready": true,
    "restartCount": 2,
    "started": true,
    "state": {
      "running": {
        "startedAt": "2023-01-31T05:05:26Z"
      }
    }
  }
]

You can increase the value of limits.memory

[JonathanGildevall]

Did all of the above, still get OOMKilled despite the memory use being well below the limit and available amount.
The memory usage seem to be climbing over time: