flux is stuck in "reconcilliation in progress" and I don't know why

[alita-moore]

my flux is stuck in reconcilliation in progress for multiple resources. It has been like this for a day. None of the logs points to any issues that I can figure out. I went through https://fluxcd.io/flux/cheatsheets/troubleshooting/ and it didn't reveal any errors worth investigating. Everything says it's ready but then the state of the kustomizations are still "Reconcilliation in progress".

Every 2.0s: kubectl get kustomizations --all-namespaces                                                                                           docker-desktop: Mon Dec 16 00:21:50 2024

NAMESPACE     NAME                          AGE     READY     STATUS
flux-system   cert-manager-system           3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   flux-system                   3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   istio-auth                    3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   istio-system                  3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   keda-system                   33h     True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   knative-serving               3d      Unknown   Reconciliation in progress
flux-system   knative-system                3d      Unknown   Reconciliation in progress
flux-system   ml-serving-convert            2d10h   Unknown   Reconciliation in progress
flux-system   ml-serving-core               3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   ml-serving-infra              2d23h   False     dependency 'flux-system/knative-serving' is not ready
flux-system   ml-serving-queue-consumers    2d8h    Unknown   Reconciliation in progress
flux-system   ml-serving-secure-converter   2d13h   True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   sealed-secrets-system         3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715

It has been like this for a day. I have tried suspending all kustomizations and then re-enabling them flux suspend kustomization --all && flux resume kustomization --all but that did noting. I've tried forcing it to update flux reconcile source git flux-system but that also did nothing. What's bizarre is that all pods are working / ready in the namespaces that claim that they're in progress.

kubectl get all -n knative-serving && kubectl get all -n knative-system && kubectl get all -n ml-serving                                                      
NAME                                        READY   STATUS      RESTARTS       AGE
pod/activator-8546f5df76-qkcnl              1/1     Running     6 (18h ago)    3d
pod/autoscaler-68fdd6cbc-8szrp              1/1     Running     6 (18h ago)    3d
pod/autoscaler-hpa-6d96447597-ftlwx         1/1     Running     12 (18h ago)   3d
pod/controller-d566c57c5-5vtp5              1/1     Running     11 (18h ago)   3d
pod/default-domain-qvqht                    0/1     Completed   0              3d
pod/net-istio-controller-6667f8d944-trpnw   1/1     Running     12 (18h ago)   3d
pod/net-istio-webhook-5988774788-gbmlf      1/1     Running     10 (18h ago)   3d
pod/webhook-77fcf884f8-wh5q9                1/1     Running     11 (18h ago)   3d

NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                   AGE
service/activator-service            ClusterIP   10.96.145.191    <none>        9090/TCP,8008/TCP,80/TCP,81/TCP,443/TCP   3d
service/autoscaler                   ClusterIP   10.96.136.13     <none>        9090/TCP,8008/TCP,8080/TCP                3d
service/autoscaler-bucket-00-of-01   ClusterIP   10.105.227.234   <none>        8080/TCP                                  3d
service/autoscaler-hpa               ClusterIP   10.103.55.118    <none>        9090/TCP,8008/TCP                         3d
service/controller                   ClusterIP   10.102.235.190   <none>        9090/TCP,8008/TCP                         3d
service/default-domain-service       ClusterIP   10.109.124.55    <none>        80/TCP                                    3d
service/net-istio-webhook            ClusterIP   10.107.0.7       <none>        9090/TCP,8008/TCP,443/TCP                 3d
service/webhook                      ClusterIP   10.110.202.36    <none>        9090/TCP,8008/TCP,443/TCP                 3d

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/activator              1/1     1            1           3d
deployment.apps/autoscaler             1/1     1            1           3d
deployment.apps/autoscaler-hpa         1/1     1            1           3d
deployment.apps/controller             1/1     1            1           3d
deployment.apps/net-istio-controller   1/1     1            1           3d
deployment.apps/net-istio-webhook      1/1     1            1           3d
deployment.apps/webhook                1/1     1            1           3d

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/activator-8546f5df76              1         1         1       3d
replicaset.apps/autoscaler-68fdd6cbc              1         1         1       3d
replicaset.apps/autoscaler-hpa-6d96447597         1         1         1       3d
replicaset.apps/controller-d566c57c5              1         1         1       3d
replicaset.apps/net-istio-controller-6667f8d944   1         1         1       3d
replicaset.apps/net-istio-webhook-5988774788      1         1         1       3d
replicaset.apps/webhook-77fcf884f8                1         1         1       3d

NAME                                            REFERENCE              TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/activator   Deployment/activator   cpu: 0%/100%   1         20        1          3d
horizontalpodautoscaler.autoscaling/webhook     Deployment/webhook     cpu: 3%/100%   1         5         1          3d

NAME                       STATUS     COMPLETIONS   DURATION   AGE
job.batch/default-domain   Complete   1/1           13s        3d
No resources found in knative-system namespace.
NAME                                                    READY   STATUS    RESTARTS       AGE
pod/convert-00017-deployment-757b5fd999-lmz7b           2/2     Running   2 (18h ago)    18h
pod/core-00017-deployment-79b4ccc5c9-9mkrr              2/2     Running   2 (18h ago)    23h
pod/minio-0                                             2/2     Running   14 (18h ago)   3d
pod/postgres-5d94df76ff-9j8z2                           2/2     Running   12 (18h ago)   3d
pod/rabbitmq-cf685cb74-9kpxg                            2/2     Running   15 (18h ago)   3d
pod/redis-69745b454c-5bnlh                              2/2     Running   12 (18h ago)   3d
pod/secure-converter-00003-deployment-cb98d8794-d6dnp   2/2     Running   8 (18h ago)    2d1h

NAME                                     TYPE           CLUSTER-IP       EXTERNAL-IP                                            PORT(S)                                              AGE
service/convert                          ExternalName   <none>           knative-local-gateway.istio-system.svc.cluster.local   80/TCP                                               2d9h
service/convert-00001                    ClusterIP      10.105.212.119   <none>                                                 80/TCP,443/TCP                                       2d9h
service/convert-00001-private            ClusterIP      10.109.133.168   <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d9h
service/convert-00002                    ClusterIP      10.105.100.232   <none>                                                 80/TCP,443/TCP                                       2d9h
service/convert-00002-private            ClusterIP      10.97.159.149    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d9h
service/convert-00003                    ClusterIP      10.98.159.222    <none>                                                 80/TCP,443/TCP                                       2d9h
service/convert-00003-private            ClusterIP      10.103.163.196   <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d9h
service/convert-00004                    ClusterIP      10.108.83.62     <none>                                                 80/TCP,443/TCP                                       2d4h
service/convert-00004-private            ClusterIP      10.105.134.234   <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d4h
service/convert-00005                    ClusterIP      10.105.142.65    <none>                                                 80/TCP,443/TCP                                       2d4h
service/convert-00005-private            ClusterIP      10.102.198.157   <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d4h
service/convert-00006                    ClusterIP      10.103.93.124    <none>                                                 80/TCP,443/TCP                                       2d4h
service/convert-00006-private            ClusterIP      10.96.104.97     <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d4h
service/convert-00007                    ClusterIP      10.96.137.240    <none>                                                 80/TCP,443/TCP                                       2d4h
service/convert-00007-private            ClusterIP      10.105.168.173   <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d4h
service/convert-00008                    ClusterIP      10.99.111.214    <none>                                                 80/TCP,443/TCP                                       2d3h
service/convert-00008-private            ClusterIP      10.103.197.89    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d3h
service/convert-00009                    ClusterIP      10.111.7.249     <none>                                                 80/TCP,443/TCP                                       29h
service/convert-00009-private            ClusterIP      10.105.228.60    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   29h
service/convert-00010                    ClusterIP      10.101.232.138   <none>                                                 80/TCP,443/TCP                                       25h
service/convert-00010-private            ClusterIP      10.97.146.229    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   25h
service/convert-00011                    ClusterIP      10.110.31.243    <none>                                                 80/TCP,443/TCP                                       24h
service/convert-00011-private            ClusterIP      10.107.89.246    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   24h
service/convert-00012                    ClusterIP      10.103.130.186   <none>                                                 80/TCP,443/TCP                                       23h
service/convert-00012-private            ClusterIP      10.102.4.109     <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   23h
service/convert-00013                    ClusterIP      10.99.9.169      <none>                                                 80/TCP,443/TCP                                       23h
service/convert-00013-private            ClusterIP      10.104.182.191   <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   23h
service/convert-00014                    ClusterIP      10.108.77.102    <none>                                                 80/TCP,443/TCP                                       20h
service/convert-00014-private            ClusterIP      10.96.151.70     <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   20h
service/convert-00015                    ClusterIP      10.105.86.177    <none>                                                 80/TCP,443/TCP                                       20h
service/convert-00015-private            ClusterIP      10.97.180.203    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   20h
service/convert-00016                    ClusterIP      10.99.138.14     <none>                                                 80/TCP,443/TCP                                       20h
service/convert-00016-private            ClusterIP      10.104.234.110   <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   20h
service/convert-00017                    ClusterIP      10.105.177.161   <none>                                                 80/TCP,443/TCP                                       18h
service/convert-00017-private            ClusterIP      10.102.97.182    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   18h
service/core                             ExternalName   <none>           knative-local-gateway.istio-system.svc.cluster.local   80/TCP                                               2d23h
service/core-00001                       ClusterIP      10.98.255.151    <none>                                                 80/TCP,443/TCP                                       2d23h
service/core-00001-private               ClusterIP      10.102.248.25    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d23h
service/core-00002                       ClusterIP      10.109.162.189   <none>                                                 80/TCP,443/TCP                                       2d13h
service/core-00002-private               ClusterIP      10.111.83.237    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d13h
service/core-00003                       ClusterIP      10.107.221.86    <none>                                                 80/TCP,443/TCP                                       2d13h
service/core-00003-private               ClusterIP      10.98.192.217    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d13h
service/core-00004                       ClusterIP      10.106.156.237   <none>                                                 80/TCP,443/TCP                                       2d10h
service/core-00004-private               ClusterIP      10.98.37.123     <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d10h
service/core-00005                       ClusterIP      10.97.212.144    <none>                                                 80/TCP,443/TCP                                       2d9h
service/core-00005-private               ClusterIP      10.104.211.80    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d9h
service/core-00006                       ClusterIP      10.108.104.117   <none>                                                 80/TCP,443/TCP                                       2d7h
service/core-00006-private               ClusterIP      10.108.208.113   <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d7h
service/core-00007                       ClusterIP      10.98.53.58      <none>                                                 80/TCP,443/TCP                                       2d7h
service/core-00007-private               ClusterIP      10.106.81.141    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d7h
service/core-00008                       ClusterIP      10.108.40.56     <none>                                                 80/TCP,443/TCP                                       2d5h
service/core-00008-private               ClusterIP      10.96.185.172    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d5h
service/core-00009                       ClusterIP      10.110.26.142    <none>                                                 80/TCP,443/TCP                                       2d5h
service/core-00009-private               ClusterIP      10.111.54.16     <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d5h
service/core-00010                       ClusterIP      10.103.241.44    <none>                                                 80/TCP,443/TCP                                       2d3h
service/core-00010-private               ClusterIP      10.108.117.247   <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d3h
service/core-00011                       ClusterIP      10.110.145.142   <none>                                                 80/TCP,443/TCP                                       2d1h
service/core-00011-private               ClusterIP      10.97.49.27      <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d1h
service/core-00012                       ClusterIP      10.106.2.98      <none>                                                 80/TCP,443/TCP                                       29h
service/core-00012-private               ClusterIP      10.102.136.156   <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   29h
service/core-00013                       ClusterIP      10.101.27.88     <none>                                                 80/TCP,443/TCP                                       25h
service/core-00013-private               ClusterIP      10.110.220.92    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   25h
service/core-00014                       ClusterIP      10.96.206.51     <none>                                                 80/TCP,443/TCP                                       25h
service/core-00014-private               ClusterIP      10.105.121.196   <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   25h
service/core-00015                       ClusterIP      10.105.168.158   <none>                                                 80/TCP,443/TCP                                       24h
service/core-00015-private               ClusterIP      10.107.28.204    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   24h
service/core-00016                       ClusterIP      10.97.10.210     <none>                                                 80/TCP,443/TCP                                       24h
service/core-00016-private               ClusterIP      10.99.144.254    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   24h
service/core-00017                       ClusterIP      10.99.191.8      <none>                                                 80/TCP,443/TCP                                       23h
service/core-00017-private               ClusterIP      10.96.12.149     <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   23h
service/minio-service                    ClusterIP      None             <none>                                                 9000/TCP,9001/TCP                                    3d
service/postgres                         ClusterIP      10.105.15.147    <none>                                                 5432/TCP                                             3d
service/rabbitmq                         ClusterIP      10.106.222.59    <none>                                                 5672/TCP,15672/TCP                                   3d
service/redis                            ClusterIP      10.104.100.57    <none>                                                 6379/TCP                                             3d
service/secure-converter                 ExternalName   <none>           knative-local-gateway.istio-system.svc.cluster.local   80/TCP                                               2d13h
service/secure-converter-00001           ClusterIP      10.108.60.37     <none>                                                 80/TCP,443/TCP                                       2d13h
service/secure-converter-00001-private   ClusterIP      10.96.11.35      <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d13h
service/secure-converter-00002           ClusterIP      10.96.40.238     <none>                                                 80/TCP,443/TCP                                       2d13h
service/secure-converter-00002-private   ClusterIP      10.102.179.38    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d13h
service/secure-converter-00003           ClusterIP      10.100.25.202    <none>                                                 80/TCP,443/TCP                                       2d1h
service/secure-converter-00003-private   ClusterIP      10.111.32.219    <none>                                                 80/TCP,443/TCP,9090/TCP,9091/TCP,8022/TCP,8012/TCP   2d1h

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/convert-00001-deployment            0/0     0            0           2d9h
deployment.apps/convert-00002-deployment            0/0     0            0           2d9h
deployment.apps/convert-00003-deployment            0/0     0            0           2d9h
deployment.apps/convert-00004-deployment            0/0     0            0           2d4h
deployment.apps/convert-00005-deployment            0/0     0            0           2d4h
deployment.apps/convert-00006-deployment            0/0     0            0           2d4h
deployment.apps/convert-00007-deployment            0/0     0            0           2d4h
deployment.apps/convert-00008-deployment            0/0     0            0           2d3h
deployment.apps/convert-00009-deployment            0/0     0            0           29h
deployment.apps/convert-00010-deployment            0/0     0            0           25h
deployment.apps/convert-00011-deployment            0/0     0            0           24h
deployment.apps/convert-00012-deployment            0/0     0            0           23h
deployment.apps/convert-00013-deployment            0/0     0            0           23h
deployment.apps/convert-00014-deployment            0/0     0            0           20h
deployment.apps/convert-00015-deployment            0/0     0            0           20h
deployment.apps/convert-00016-deployment            0/0     0            0           20h
deployment.apps/convert-00017-deployment            1/1     1            1           18h
deployment.apps/core-00001-deployment               0/0     0            0           2d23h
deployment.apps/core-00002-deployment               0/0     0            0           2d13h
deployment.apps/core-00003-deployment               0/0     0            0           2d13h
deployment.apps/core-00004-deployment               0/0     0            0           2d10h
deployment.apps/core-00005-deployment               0/0     0            0           2d9h
deployment.apps/core-00006-deployment               0/0     0            0           2d7h
deployment.apps/core-00007-deployment               0/0     0            0           2d7h
deployment.apps/core-00008-deployment               0/0     0            0           2d5h
deployment.apps/core-00009-deployment               0/0     0            0           2d5h
deployment.apps/core-00010-deployment               0/0     0            0           2d3h
deployment.apps/core-00011-deployment               0/0     0            0           2d1h
deployment.apps/core-00012-deployment               0/0     0            0           29h
deployment.apps/core-00013-deployment               0/0     0            0           25h
deployment.apps/core-00014-deployment               0/0     0            0           25h
deployment.apps/core-00015-deployment               0/0     0            0           24h
deployment.apps/core-00016-deployment               0/0     0            0           24h
deployment.apps/core-00017-deployment               1/1     1            1           23h
deployment.apps/postgres                            1/1     1            1           3d
deployment.apps/rabbitmq                            1/1     1            1           3d
deployment.apps/redis                               1/1     1            1           3d
deployment.apps/secure-converter-00001-deployment   0/0     0            0           2d13h
deployment.apps/secure-converter-00002-deployment   0/0     0            0           2d13h
deployment.apps/secure-converter-00003-deployment   1/1     1            1           2d1h

NAME                                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/convert-00001-deployment-59b9b9f489            0         0         0       2d9h
replicaset.apps/convert-00002-deployment-7d59c559b4            0         0         0       2d9h
replicaset.apps/convert-00003-deployment-7b6789d4f6            0         0         0       2d9h
replicaset.apps/convert-00004-deployment-6965c74fdd            0         0         0       2d4h
replicaset.apps/convert-00005-deployment-7859fdd798            0         0         0       2d4h
replicaset.apps/convert-00006-deployment-7f64fff767            0         0         0       2d4h
replicaset.apps/convert-00007-deployment-b85857b8b             0         0         0       2d4h
replicaset.apps/convert-00008-deployment-56856b99c7            0         0         0       2d3h
replicaset.apps/convert-00009-deployment-654d865c89            0         0         0       29h
replicaset.apps/convert-00010-deployment-c5849dc95             0         0         0       25h
replicaset.apps/convert-00011-deployment-c9c48fcf8             0         0         0       24h
replicaset.apps/convert-00012-deployment-b55464865             0         0         0       23h
replicaset.apps/convert-00013-deployment-6d9cf7845c            0         0         0       23h
replicaset.apps/convert-00014-deployment-5cf47f8bff            0         0         0       20h
replicaset.apps/convert-00015-deployment-86d7ccffc8            0         0         0       20h
replicaset.apps/convert-00016-deployment-5486b57597            0         0         0       20h
replicaset.apps/convert-00017-deployment-757b5fd999            1         1         1       18h
replicaset.apps/core-00001-deployment-6bfcd5c55                0         0         0       2d23h
replicaset.apps/core-00002-deployment-6946bcc7c8               0         0         0       2d13h
replicaset.apps/core-00003-deployment-69b7f8d64f               0         0         0       2d10h
replicaset.apps/core-00003-deployment-6bbb774d68               0         0         0       2d13h
replicaset.apps/core-00004-deployment-6fbc8dff48               0         0         0       2d10h
replicaset.apps/core-00005-deployment-846fc4cf78               0         0         0       2d9h
replicaset.apps/core-00006-deployment-59867bcb4b               0         0         0       2d7h
replicaset.apps/core-00007-deployment-7bdcfb7f9c               0         0         0       2d7h
replicaset.apps/core-00008-deployment-dc8cccdd6                0         0         0       2d5h
replicaset.apps/core-00009-deployment-76f846b78b               0         0         0       2d5h
replicaset.apps/core-00010-deployment-6c59b5f97                0         0         0       2d3h
replicaset.apps/core-00011-deployment-58bf85d66d               0         0         0       2d1h
replicaset.apps/core-00012-deployment-6b9544654c               0         0         0       29h
replicaset.apps/core-00013-deployment-6c86dd9954               0         0         0       25h
replicaset.apps/core-00014-deployment-6c8cbbff98               0         0         0       25h
replicaset.apps/core-00015-deployment-657b7b8474               0         0         0       24h
replicaset.apps/core-00016-deployment-685974d9c4               0         0         0       24h
replicaset.apps/core-00017-deployment-79b4ccc5c9               1         1         1       23h
replicaset.apps/postgres-5d94df76ff                            1         1         1       3d
replicaset.apps/rabbitmq-cf685cb74                             1         1         1       3d
replicaset.apps/redis-69745b454c                               1         1         1       3d
replicaset.apps/secure-converter-00001-deployment-676f884b5d   0         0         0       2d13h
replicaset.apps/secure-converter-00002-deployment-75654bdb4d   0         0         0       2d13h
replicaset.apps/secure-converter-00003-deployment-cb98d8794    1         1         1       2d1h

NAME                     READY   AGE
statefulset.apps/minio   1/1     3d

NAME                                                                        REFERENCE                    TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/keda-hpa-queue-consumers-scaledobject   Deployment/queue-consumers   0/5 (avg)   1         10        1          32h

NAME                                                 LATESTCREATED            LATESTREADY              READY   REASON
configuration.serving.knative.dev/convert            convert-00017            convert-00017            True    
configuration.serving.knative.dev/core               core-00017               core-00017               True    
configuration.serving.knative.dev/secure-converter   secure-converter-00003   secure-converter-00003   True    

NAME                                                  CONFIG NAME        GENERATION   READY   REASON                     ACTUAL REPLICAS   DESIRED REPLICAS
revision.serving.knative.dev/convert-00001            convert            1            True                               0                 0
revision.serving.knative.dev/convert-00002            convert            2            True                               0                 0
revision.serving.knative.dev/convert-00003            convert            3            True                               0                 0
revision.serving.knative.dev/convert-00004            convert            4            True                               0                 0
revision.serving.knative.dev/convert-00005            convert            5            True                               0                 0
revision.serving.knative.dev/convert-00006            convert            6            True                               0                 0
revision.serving.knative.dev/convert-00007            convert            7            True                               0                 0
revision.serving.knative.dev/convert-00008            convert            8            True                               0                 0
revision.serving.knative.dev/convert-00009            convert            9            True                               0                 0
revision.serving.knative.dev/convert-00010            convert            10           False   ExitCode137                0                 0
revision.serving.knative.dev/convert-00011            convert            11           True                               0                 0
revision.serving.knative.dev/convert-00012            convert            12           True                               0                 0
revision.serving.knative.dev/convert-00013            convert            13           True                               0                 0
revision.serving.knative.dev/convert-00014            convert            14           True                               0                 0
revision.serving.knative.dev/convert-00015            convert            15           True                               0                 0
revision.serving.knative.dev/convert-00016            convert            16           True                               0                 0
revision.serving.knative.dev/convert-00017            convert            17           True                               1                 1
revision.serving.knative.dev/core-00001               core               1            True                               0                 0
revision.serving.knative.dev/core-00002               core               2            False   ProgressDeadlineExceeded   0                 0
revision.serving.knative.dev/core-00003               core               3            False   ExitCode255                0                 0
revision.serving.knative.dev/core-00004               core               4            True                               0                 0
revision.serving.knative.dev/core-00005               core               5            True                               0                 0
revision.serving.knative.dev/core-00006               core               6            True                               0                 0
revision.serving.knative.dev/core-00007               core               7            True                               0                 0
revision.serving.knative.dev/core-00008               core               8            True                               0                 0
revision.serving.knative.dev/core-00009               core               9            True                               0                 0
revision.serving.knative.dev/core-00010               core               10           True                               0                 0
revision.serving.knative.dev/core-00011               core               11           False   ExitCode1                  0                 0
revision.serving.knative.dev/core-00012               core               12           True                               0                 0
revision.serving.knative.dev/core-00013               core               13           True                               0                 0
revision.serving.knative.dev/core-00014               core               14           False   ExitCode137                0                 0
revision.serving.knative.dev/core-00015               core               15           True                               0                 0
revision.serving.knative.dev/core-00016               core               16           True                               0                 0
revision.serving.knative.dev/core-00017               core               17           True                               1                 1
revision.serving.knative.dev/secure-converter-00001   secure-converter   1            False   ProgressDeadlineExceeded   0                 0
revision.serving.knative.dev/secure-converter-00002   secure-converter   2            True                               0                 0
revision.serving.knative.dev/secure-converter-00003   secure-converter   3            True                               1                 1

NAME                                         URL                                              READY   REASON
route.serving.knative.dev/convert            http://convert.ml-serving.example.com            True    
route.serving.knative.dev/core               http://core.ml-serving.example.com               True    
route.serving.knative.dev/secure-converter   http://secure-converter.ml-serving.example.com   True    

NAME                                           URL                                              LATESTCREATED            LATESTREADY              READY   REASON
service.serving.knative.dev/convert            http://convert.ml-serving.example.com            convert-00017            convert-00017            True    
service.serving.knative.dev/core               http://core.ml-serving.example.com               core-00017               core-00017               True    
service.serving.knative.dev/secure-converter   http://secure-converter.ml-serving.example.com   secure-converter-00003   secure-converter-00003   True  

I'm out of ideas on what to try next. Also note that the applied revision is not up to date, the latest sha is different. This is known by flux / it claims it has reconcilled the git source yet the system is not propogating this.

NOTE: this is NOT production, but I'd like to figure out what's going on so in case it ever happens in production

[alita-moore]

amazingly, I can't even force delete the namespace / kustomizations. I wanted to try reinstalling flux

➜  monorepo git:(main) kubectl get ns                                            
NAME                    STATUS        AGE
cert-manager            Active        3d
default                 Active        3d
flux-system             Terminating   3d
istio-system            Active        3d
keda-system             Active        33h
knative-serving         Active        3d
kube-node-lease         Active        3d
kube-public             Active        3d
kube-system             Active        3d
kubernetes-dashboard    Active        18h
ml-serving              Active        3d
sealed-secrets-system   Active        3d
➜  monorepo git:(main) kubectl get all -n flux-system                                                                          
No resources found in flux-system namespace.
➜  monorepo git:(main) kubectl get kustomizations.kustomize.toolkit.fluxcd.io -A
NAMESPACE     NAME                          AGE     READY     STATUS
flux-system   cert-manager-system           3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   flux-system                   3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   istio-auth                    3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   istio-system                  3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   keda-system                   33h     True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   knative-serving               3d      Unknown   Reconciliation in progress
flux-system   knative-system                3d      Unknown   Reconciliation in progress
flux-system   ml-serving-convert            2d10h   Unknown   Reconciliation in progress
flux-system   ml-serving-core               3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   ml-serving-infra              3d      False     dependency 'flux-system/knative-serving' is not ready
flux-system   ml-serving-queue-consumers    2d8h    Unknown   Reconciliation in progress
flux-system   ml-serving-secure-converter   2d13h   True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   sealed-secrets-system         3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
➜  monorepo git:(main) kubectl get ns                                           
NAME                    STATUS        AGE
cert-manager            Active        3d
default                 Active        3d
flux-system             Terminating   3d
istio-system            Active        3d
keda-system             Active        33h
knative-serving         Active        3d
kube-node-lease         Active        3d
kube-public             Active        3d
kube-system             Active        3d
kubernetes-dashboard    Active        18h
ml-serving              Active        3d
sealed-secrets-system   Active        3d
➜  monorepo git:(main) kubectl get namespace flux-system -o json | jq '.spec.finalizers'
[
  "kubernetes"
]
➜  monorepo git:(main) kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --namespace flux-system --ignore-not-found
NAME                          AGE     READY     STATUS
cert-manager-system           3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system                   3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
istio-auth                    3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
istio-system                  3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
keda-system                   33h     True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
knative-serving               3d      Unknown   Reconciliation in progress
knative-system                3d      Unknown   Reconciliation in progress
ml-serving-convert            2d10h   Unknown   Reconciliation in progress
ml-serving-core               3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
ml-serving-infra              3d      False     dependency 'flux-system/knative-serving' is not ready
ml-serving-queue-consumers    2d8h    Unknown   Reconciliation in progress
ml-serving-secure-converter   2d13h   True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
sealed-secrets-system         3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
➜  monorepo git:(main) kubectl get all -n flux-system                                                                                             
No resources found in flux-system namespace.
➜  monorepo git:(main) kubectl delete ns flux-system --force
Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
namespace "flux-system" force deleted
^C%                                                                                                                                                                                       
➜  monorepo git:(main) kubectl get ns                                                   
NAME                    STATUS        AGE
cert-manager            Active        3d
default                 Active        3d
flux-system             Terminating   3d
istio-system            Active        3d
keda-system             Active        33h
knative-serving         Active        3d
kube-node-lease         Active        3d
kube-public             Active        3d
kube-system             Active        3d
kubernetes-dashboard    Active        18h
ml-serving              Active        3d
sealed-secrets-system   Active        3d
➜  monorepo git:(main) kubectl get kustomizations -A                            
NAMESPACE     NAME                          AGE     READY     STATUS
flux-system   cert-manager-system           3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   flux-system                   3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   istio-auth                    3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   istio-system                  3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   keda-system                   33h     True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   knative-serving               3d      Unknown   Reconciliation in progress
flux-system   knative-system                3d      Unknown   Reconciliation in progress
flux-system   ml-serving-convert            2d10h   Unknown   Reconciliation in progress
flux-system   ml-serving-core               3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   ml-serving-infra              3d      False     dependency 'flux-system/knative-serving' is not ready
flux-system   ml-serving-queue-consumers    2d8h    Unknown   Reconciliation in progress
flux-system   ml-serving-secure-converter   2d13h   True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system   sealed-secrets-system         3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
➜  monorepo git:(main) kubectl get nskubectl delete kustomizations --all -n flux-system --force --grace-period=0
error: unknown flag: --all
See 'kubectl get --help' for usage.
➜  monorepo git:(main) kubectl delete kustomizations --all -n flux-system --force --grace-period=0
Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
kustomization.kustomize.toolkit.fluxcd.io "cert-manager-system" force deleted
kustomization.kustomize.toolkit.fluxcd.io "flux-system" force deleted
kustomization.kustomize.toolkit.fluxcd.io "istio-auth" force deleted
kustomization.kustomize.toolkit.fluxcd.io "istio-system" force deleted
kustomization.kustomize.toolkit.fluxcd.io "keda-system" force deleted
kustomization.kustomize.toolkit.fluxcd.io "knative-serving" force deleted
kustomization.kustomize.toolkit.fluxcd.io "knative-system" force deleted
kustomization.kustomize.toolkit.fluxcd.io "ml-serving-convert" force deleted
kustomization.kustomize.toolkit.fluxcd.io "ml-serving-core" force deleted
kustomization.kustomize.toolkit.fluxcd.io "ml-serving-infra" force deleted
kustomization.kustomize.toolkit.fluxcd.io "ml-serving-queue-consumers" force deleted
kustomization.kustomize.toolkit.fluxcd.io "ml-serving-secure-converter" force deleted
kustomization.kustomize.toolkit.fluxcd.io "sealed-secrets-system" force deleted
^C%                                                                                                                                                                                       
➜  monorepo git:(main) kubectl get kustomizations -n flux-system
NAME                          AGE     READY     STATUS
cert-manager-system           3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system                   3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
istio-auth                    3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
istio-system                  3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
keda-system                   33h     True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
knative-serving               3d      Unknown   Reconciliation in progress
knative-system                3d      Unknown   Reconciliation in progress
ml-serving-convert            2d10h   Unknown   Reconciliation in progress
ml-serving-core               3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
ml-serving-infra              3d      False     dependency 'flux-system/knative-serving' is not ready
ml-serving-queue-consumers    2d8h    Unknown   Reconciliation in progress
ml-serving-secure-converter   2d13h   True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
sealed-secrets-system         3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
➜  monorepo git:(main) kubectl patch kustomizations -n flux-system --all -p '{"spec": {"suspend": true}}' --type=merge
error: unknown flag: --all
See 'kubectl patch --help' for usage.
➜  monorepo git:(main) kubectl delete kustomizations --all -n flux-system --force --grace-period=0
Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
kustomization.kustomize.toolkit.fluxcd.io "cert-manager-system" force deleted
kustomization.kustomize.toolkit.fluxcd.io "flux-system" force deleted
kustomization.kustomize.toolkit.fluxcd.io "istio-auth" force deleted
kustomization.kustomize.toolkit.fluxcd.io "istio-system" force deleted
kustomization.kustomize.toolkit.fluxcd.io "keda-system" force deleted
kustomization.kustomize.toolkit.fluxcd.io "knative-serving" force deleted
kustomization.kustomize.toolkit.fluxcd.io "knative-system" force deleted
kustomization.kustomize.toolkit.fluxcd.io "ml-serving-convert" force deleted
kustomization.kustomize.toolkit.fluxcd.io "ml-serving-core" force deleted
kustomization.kustomize.toolkit.fluxcd.io "ml-serving-infra" force deleted
kustomization.kustomize.toolkit.fluxcd.io "ml-serving-queue-consumers" force deleted
kustomization.kustomize.toolkit.fluxcd.io "ml-serving-secure-converter" force deleted
kustomization.kustomize.toolkit.fluxcd.io "sealed-secrets-system" force deleted
^C%                                                                                                                                                                                       
➜  monorepo git:(main) kubectl delete pod -n flux-system --all --force --grace-period=0
Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
No resources found
➜  monorepo git:(main) kubectl get kustomizations -n flux-system                       
NAME                          AGE     READY     STATUS
cert-manager-system           3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
flux-system                   3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
istio-auth                    3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
istio-system                  3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
keda-system                   33h     True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
knative-serving               3d      Unknown   Reconciliation in progress
knative-system                3d      Unknown   Reconciliation in progress
ml-serving-convert            2d10h   Unknown   Reconciliation in progress
ml-serving-core               3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
ml-serving-infra              3d      False     dependency 'flux-system/knative-serving' is not ready
ml-serving-queue-consumers    2d8h    Unknown   Reconciliation in progress
ml-serving-secure-converter   2d13h   True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
sealed-secrets-system         3d      True      Applied revision: main@sha1:37329bdfb1ff60c0f7b9c6e81f6bc52849b52715
➜  monorepo git:(main) kubectl get ns flux-system
NAME          STATUS        AGE
flux-system   Terminating   3d

I tried to force re-bootstrap the system and go the following:

flux bootstrap github \
  --owner=wovenmind \
  --repository=monorepo \
  --branch=main \
  --path=./clusters/local
Please enter your GitHub personal access token (PAT): 
► connecting to github.com
► cloning branch "main" from Git repository "https://github.com/wovenmind/monorepo.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ committed component manifests to "main" ("8eb1e7708e21c14bfd47f12ce5179eb1a925ab5d")
► pushing component manifests to "https://github.com/wovenmind/monorepo.git"
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
► generating source secret
✔ public key: ecdsa-sha2-nistp384 
✔ configured deploy key "flux-system-main-flux-system-./clusters/local" for "https://github.com/wovenmind/monorepo"
► applying source secret "flux-system/flux-system"
✗ secrets "flux-system" is forbidden: unable to create new content in namespace flux-system because it is being terminated

[alita-moore]

I was able to get it to uninstall by running flux uninstall which then allowed me to run flux bootstrap which seems to have force reset the system and allowed things to get unstuck. I also tried restarting the cluster so it's hard to say the effect on the cluster if this were production. But I think it's probably safe to do this in production as well.. (but it would be nice if a maintainer could verify that doing this would not directly affect / shutdown the managed resources)