Flux stuck on old commit

[psammarco]

Hello,

I am new to flux and I am trying to figure out the reason why is not picking up new commits.

It seems to be stuck on a fairly old commit, caf847c0d86ff575025979fe18ec2799f859b9d8.

Flux is deployed on my k3s homelab and was bootstrapped following the GitLab PAT documentation. Flux GitOps GitLab repo here.

Flux version: 2.4.0
k3s version: v1.29.6+k3s2 (b4b156d9)

admin@store0:~$ flux reconcile source git flux-system --namespace flux-system
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision main@sha1:caf847c0d86ff575025979fe18ec2799f859b9d8
admin@store0:~$ flux reconcile kustomization flux-system --namespace flux-system
► annotating Kustomization flux-system in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ applied revision main@sha1:caf847c0d86ff575025979fe18ec2799f859b9d8

admin@store0:~$ kubectl get gitrepository flux-system -n flux-system -o yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  annotations:
    reconcile.fluxcd.io/requestedAt: "2024-12-17T20:12:04.29094633Z"
  creationTimestamp: "2024-12-11T18:31:46Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 1
  labels:
    kustomize.toolkit.fluxcd.io/name: flux-system
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: flux-system
  namespace: flux-system
  resourceVersion: "23771394"
  uid: 436bf699-1bb3-4a85-94b4-2c6cfc0cee83
spec:
  interval: 1m0s
  ref:
    branch: main
  secretRef:
    name: flux-system
  timeout: 60s
  url: ssh://[email protected]/kernel.panic007/flux-cluster.git
status:
  artifact:
    digest: sha256:b686577e7f092d5c5613af6e6b458c4ed3b36eceb15c59a7e622accf081733af
    lastUpdateTime: "2024-12-11T18:28:01Z"
    path: gitrepository/flux-system/flux-system/caf847c0d86ff575025979fe18ec2799f859b9d8.tar.gz
    revision: main@sha1:caf847c0d86ff575025979fe18ec2799f859b9d8
    size: 68690
    url: http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/caf847c0d86ff575025979fe18ec2799f859b9d8.tar.gz
  conditions:
  - lastTransitionTime: "2024-12-12T05:56:48Z"
    message: stored artifact for revision 'main@sha1:caf847c0d86ff575025979fe18ec2799f859b9d8'
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: Ready
  - lastTransitionTime: "2024-12-11T18:28:01Z"
    message: stored artifact for revision 'main@sha1:caf847c0d86ff575025979fe18ec2799f859b9d8'
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: ArtifactInStorage
  lastHandledReconcileAt: "2024-12-17T20:12:04.29094633Z"
  observedGeneration: 1

Flux resources on k3s cluster.

admin@store0:~$ kubectl get pods -n flux-system
NAME                                           READY   STATUS    RESTARTS      AGE
helm-controller-7d4bd79b4-sljf5                1/1     Running   4 (9d ago)    46d
image-automation-controller-77f68fcf95-tj7l7   1/1     Running   7 (9d ago)    46d
image-reflector-controller-697d66dd6d-vx5x9    1/1     Running   7 (9d ago)    46d
kustomize-controller-85c747587c-fwh6b          1/1     Running   10 (9d ago)   46d
notification-controller-6c8d8b9b8f-xb59x       1/1     Running   7 (9d ago)    46d
source-controller-5f8d567fcd-8vrc8             1/1     Running   10 (9d ago)   46d
admin@store0:~$ kubectl get deployments -n flux-system
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
helm-controller               1/1     1            1           46d
image-automation-controller   1/1     1            1           46d
image-reflector-controller    1/1     1            1           46d
kustomize-controller          1/1     1            1           46d
notification-controller       1/1     1            1           46d
source-controller             1/1     1            1           46d
admin@store0:~$

I am more than happy to go through the Flux policies and CI if necessary. I am trying to avoid providing too much unnecessary info.

Thanks

[stefanprodan]

It seems to be stuck on a fairly old commit, caf847c0d86ff575025979fe18ec2799f859b9d8.

This is the latest commit to your main brach https://gitlab.com/kernel.panic007/flux-cluster/-/commits/main?ref_type=heads

[psammarco]

Thanks for your response and apology for getting back to you this late. I have been unwell.

You are absolutely right, that is the last commit and I was wrong in thinking Flux was stuck on the initial commit. For some reason I got things mixed up during my initial troubleshooting of the issue.

At this point I can only guess I must be doing something wrong, being with the configuration or design. Let me give you a brief of what I am trying to do.

Basically I have setup flux to track buildah image tag, and according to my understanding of the setup and design, I am expecting Flux to update flux-cluster/clusters/my-cluster/flux-system/image-tags/buildah.tag when a new image tag is found. This does not happen.

This is image-policies/buildah.yaml for the image tracking

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: buildah-latest
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: buildah
  filterTags:
    pattern: '^v1\..*' # because immage tag in question is v1.x
  policy:
    semver:
      range: "1.x"

and image-repositories/buildah.yaml

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: buildah
  namespace: flux-system
spec:
  interval: 1h
  image: quay.io/buildah/stable

Lastly, image-automations/buildah.yaml should take care of updating my repo whenever a new image tag for buildah is found

kind: ImageUpdateAutomation
metadata:
  name: buildah-automation
  namespace: flux-system
spec:
  interval: 1m
  sourceRef:
    kind: GitRepository
    name: flux-cluster
  git:
    commit:
      author:
        email: "[email protected]"
        name: "Automation Bot"
      messageTemplate: "Update buildah image to {{range .Updated.Images}}{{.NewTag}}{{end}}"
  update:
    path: "clusters/my-cluster/flux-system/image-tags/buildah.tag"
    strategy: Setters

From the logs I can see kustomization is working and whenever a new buildah image tag is pushed, this is picked up by the Flux cluster. Despite of that, clusters/my-cluster/flux-system/image-tags/buildah.tag is not being updated, and so my pipeline doesn't kick in.

Hopefully I explained it clear enough. Btw, Merry Christmas and Happy New year :).

[stefanprodan]

Flux only updates Kubernetes YAML, the path must be a dir and inside the dir you must have either a Kubernetes Deployment or some custom resource, but it needs to be Kubernetes object in YAML format. Docs here: https://fluxcd.io/flux/guides/image-update/#configure-image-update-for-custom-resources

[psammarco]

Now that makes sense. Though, I am still struggling to get this thing to work.

I changed clusters/my-cluster/flux-system/image-tags/buildah.tag to clusters/my-cluster/flux-system/image-tags/buildah.yaml and to a ConfigMap to store the value of the buildah image tag. Still, it is not getting updated.

Both imageAutomation yaml and CI files were changed accordingly to reflect the ConfigMap implementation.

As mentioned previously, Flux does detect the latest buildah tag

admin@store0:~$ kubectl logs -n flux-system -l app=image-reflector-controller |tail -1
{"level":"info","ts":"2024-12-29T21:04:24.552Z","msg":"Latest image tag for 'quay.io/buildah/stable' updated from v1.37.5 to v1.38.0","controller":"imagepolicy","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImagePolicy","ImagePolicy":{"name":"buildah-latest","namespace":"flux-system"},"namespace":"flux-system","name":"buildah-latest","reconcileID":"82168385-298d-433f-9f4e-7d9b024d379f"}
admin@store0:~$

Reconciled manually after creating the ConfigMap resource

admin@store0:~$ kubectl logs -n flux-system -l app=kustomize-controller |tail -1
{"level":"info","ts":"2024-12-29T21:03:20.210Z","msg":"Reconciliation finished in 30.969576888s, next run in 10m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"dc15fac3-681c-4c0b-a4df-20e6b885b790","revision":"main@sha1:21303e124ca0331e2400398d53f943e1278cc45d"}
admin@store0:~$

I have noticed the notification-controller has never picked up a notification. I am not sure if this is relevant to the issue.

{"level":"info","ts":"2024-12-29T21:03:20.272Z","logger":"event-server","msg":"discarding event, no alerts found for the involved object","eventInvolvedObject":{"kind":"Kustomization","namespace":"flux-system","name":"flux-system","uid":"a76e4127-5a66-4437-ba33-6241beff022c","apiVersion":"kustomize.toolkit.fluxcd.io/v1","resourceVersion":"25863991"}}

I would appreciate if you could point me to what I am doing wrong and if you could help me with an example of what the solution may look like.

Thanks a lot.

[stefanprodan]

See the docs again, you're missing the marker that tells Flux which policy to apply to which line inside the YAML https://fluxcd.io/flux/guides/image-update/#configure-image-update-for-custom-resources