Flux controllers keeps on restarting

[SamarSidharth]

I have deployed flux and it keeps on restarting after few minutes. One point to note is that there are reconcilation errors for known reasons. In the controller logs I see errors and related to reconcilations and then see a shutdown message on the controleer and this triggers restart of all the controller pods.

Any pointer to troubleshoot this issue can help. Thanks

[kingdonb]

Can you post any of the logs? There really isn't enough information to help with debugging here.

There are several controllers, which is shutting down? Is there a failure of leader election, networking, storage, or not enough RAM?
What type of cluster is used? Where is it hosted? Provide more information so we can help you find the issue please.

[SamarSidharth]

hello @kingdonb thanks for your reply.

All the controllers restart at the same time.. I am not sure which is the one that triggers it but from the logs I assume it is kustomize controller. The system has sufficient resources. It a kubernetes cluster deployed via kubeadm. It is hosted on VMs

NAME                                      READY   STATUS        RESTARTS   AGE
helm-controller-5f964c6579-nr2zt          1/1     Running       0          17s
kustomize-controller-789c87cb95-58whm     1/1     Terminating   0          64s
kustomize-controller-9c588946c-p475j      1/1     Running       0          17s
notification-controller-76dc5d768-s2wpq   1/1     Running       0          17s
source-controller-6c49485888-5n77b        1/1     Running       0          16s
root@vm1:/home/ubuntu

Kustomize controller logs

{"level":"info","ts":"2023-11-23T13:21:44.166Z","msg":"server-side apply completed","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"69ec0672-ae8d-476a-b763-724d977c1f3d","output":{"ClusterRole/crd-controller-flux-system":"configured","ClusterRole/flux-edit-flux-system":"configured","ClusterRole/flux-view-flux-system":"configured","ClusterRoleBinding/cluster-reconciler-flux-system":"configured","ClusterRoleBinding/crd-controller-flux-system":"configured","Deployment/flux-system/helm-controller":"configured","Deployment/flux-system/kustomize-controller":"configured","Deployment/flux-system/notification-controller":"configured","Deployment/flux-system/source-controller":"configured","GitRepository/flux-system/flux-system":"unchanged","Kustomization/flux-system/flux-system":"unchanged","NetworkPolicy/flux-system/allow-egress":"configured","NetworkPolicy/flux-system/allow-scraping":"configured","NetworkPolicy/flux-system/allow-webhooks":"configured","ResourceQuota/flux-system/critical-pods-flux-system":"configured","Service/flux-system/notification-controller":"configured","Service/flux-system/source-controller":"configured","Service/flux-system/webhook-receiver":"configured","ServiceAccount/flux-system/helm-controller":"configured","ServiceAccount/flux-system/kustomize-controller":"configured","ServiceAccount/flux-system/notification-controller":"configured","ServiceAccount/flux-system/source-controller":"configured"},"revision":"main@sha1:36f0dace226e79a9706ac2d9c9b0008cfe517dcd"}
{"level":"info","ts":"2023-11-23T13:21:44.230Z","msg":"Reconciliation finished in 3.577875107s, next run in 10m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"69ec0672-ae8d-476a-b763-724d977c1f3d","revision":"main@sha1:36f0dace226e79a9706ac2d9c9b0008cfe517dcd"}
{"level":"info","ts":"2023-11-23T13:21:46.392Z","msg":"Stopping and waiting for non leader election runnables"}
{"level":"info","ts":"2023-11-23T13:21:46.392Z","msg":"shutting down server","path":"/metrics","kind":"metrics","addr":"[::]:8080"}
{"level":"info","ts":"2023-11-23T13:21:46.392Z","msg":"Stopping and waiting for leader election runnables"}
{"level":"info","ts":"2023-11-23T13:21:46.392Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization"}
{"level":"info","ts":"2023-11-23T13:22:16.392Z","msg":"Stopping and waiting for caches"}
{"level":"info","ts":"2023-11-23T13:22:16.392Z","msg":"Stopping and waiting for webhooks"}
{"level":"info","ts":"2023-11-23T13:22:16.393Z","msg":"Wait completed, proceeding to shutdown the manager"}
{"level":"error","ts":"2023-11-23T13:22:16.400Z","logger":"setup","msg":"problem running manager","error":"failed waiting for all runnables to end within grace period of 30s: context deadline exceeded"}
root@vm1:/home/ubuntu#```

[SamarSidharth]

There is only a single instance of flux here, it is just that we bootstrapped it multiple times due to the restart issue.

[stefanprodan]

You also must disable that goldilocks, it can't work with Flux nor GitOps.

[SamarSidharth]

Sure

[stefanprodan]

metadata.labels.kustomize.toolkit.fluxcd.io/name
± value change

• flux

• flux-system

This tells me that there are 2 Flux instances on your cluster in the flux and flux-system namespace.

[stefanprodan]

These 2 instances are fighting each other, the version flips from Flux v2.1.1 to v2.1.2, so controllers get downgraded/upgraded in a loop.

[SamarSidharth]

Thanks a lot @stefanprodan for your help and support