Error when imageUpdateAutomation triggers an update

[2fst4u]

This isn't quite a bug because I'm sure it's something I'm either doing wrong or can fix in my setup rather than a problem with flux itself so I'm asking here.

I get the following error when an imageUpdateAutomation triggers an update on my repo:

command error on refs/heads/main: cannot lock ref ''refs/heads/main'': is at 6162cc076f884b804a018cf5b8a098c1307db330 but expected fc7f071ffce40b73b736876914be4ca907b03586'

It pops up for a handful of different imageUpdateAutomations each time and I only see it happen when another object's tag gets updated by an automation. Nothing breaks, everything works fine but it's messy and I want to know if there's anything I can do to resolve it.

I'm quite new to git and research online shows a prune can fix this if you're using git as an individual and you get this but I don't know if flux works the same way.

Any help appreciated.

[kingdonb]

In the last release or very recent release, we moved from libgit2 to go-git. This is a major change and it will help to know what version of Flux you are running (please paste the entire output of flux check) to determine if this is an issue with the new change, or if it's an issue that might have already been resolved in the new version.

The change was in the release on November 22, that's Flux 0.27.x 0.37.0, which removed libgit2

(If you're upgrading, we recommend to go with the latest version, some other issues that were reported have been fixed in 0.37)

[2fst4u]

#3443

This is the thread here where we figured out what I did wrong the first time.

After bootstrapping again, I could never figure out the correct way to add decryption, so I now have it added to one of the flux created yaml files you mentioned to get it working, so as you say it will be deleted.

I could never figure out how to add notification provers through cli, so those are in there as well.

I'd love to have both of those added the "correct" way so they don't get broken by updates.

[kingdonb]

There are two pieces of advice I can give that will hopefully clarify how this can work

1. when you put files into clusters/my-cluster/ if you ONLY put Flux Kustomizations there, this is the advice for best practices

2. the simplest way to explain the best practice for handling secrets is that you should aim to keep them in a separate repository that has more restricted access than the non-secrets. This advice has a big asterisk though, as you may have already found – I'm not sure what blocks you from implementing this advice from the other thread, but at a guess I have some ideas, and I'll elaborate:

If you are going to use Kustomize's secretGenerator (which tl;dr is something you must do, if you want HelmReleases to work properly with encrypted secrets as input, reference) then you actually need the secret to be located together with the helmrelease, so the secret can be hashed and included at apply time. This advice is flatly in conflict with (2) unless you have set up your repo structure very carefully, and/or use some advanced Flux features like GitRepository.spec.includes.

Given this constraint, you might not really want to keep secrets in a separate git repository, includes is frankly a complicated feature and I don't recommend it except as a last resort.

Generally if I was to try to provide easier advice to follow that might help solve this for you, it would be this:

(1) don't put anything else inside of clusters/my-cluster/flux-system but if you must put it there, do not put it directly into any of the files that Flux creates gotk-*.yaml – put the files alongside, and list them in kustomization.yaml. But you can skip that step if...

(2) ...instead, put them in neighboring directories alongside of flux-system. You can make flux-system-extras for example, and put in there things like the YAML definition for your notification providers etc. Those things like notification provider that depend on a secret won't work until the secret is applied, but that's OK, because flux-system does not typically set spec.wait to avoid blocking on itself or other Flux kustomizations downstream. And that's...

(3) where you should be applying your secrets from, "other Flux kustomizations downstream" of flux-system.

Stefan's advice was to only use the flux-system Kustomization to apply other Flux Kustomizations. Don't use it to apply secrets, or HelmReleases, or deployments or other workloads.

Here is an example, of how I have done it on my cluster (moo-cluster):
https://github.com/kingdonb/bootstrap-repo/blob/main/clusters/moo-cluster/my-secrets/my-secrets-kustomization.yaml – I have mostly followed this advice, but there are definitely some resources in there breaking this rule. Still, as a general rule it's mostly adhered to. All of the workloads should be applied "in their own Flux Kustomization" separate from Flux's own.

You can set up one Flux kustomization that applies all the secrets, to keep them separate and confine the use of the decryption key to avoid compromising it inside of a tenant, who should only have access to a decrypted secret and not the decryption key... or you can let tenants manage their own keys, (which is not something I was prepared to do, I'd rather just centralize all the secrets, and I'm mostly comfortable with the compromises that come from not being able to use secretGenerator with the secrets, because they are not located together with those artifacts that consume them.)

In the Kustomization that applies secrets, I point it at a directory in the same repo:
https://github.com/kingdonb/bootstrap-repo/blob/main/clusters/moo-cluster/my-secrets/my-secrets-kustomization.yaml#L17

There are probably multiple things wrong with my example, but this one I want to highlight. My example repo is public, and the encrypted secrets are public: do not follow my example here! There is no reason to make your infrastructure repo public, but there is really no reason to make your encrypted secrets public.

I should not do this, I'm just not sure how to get the example across clearly with some repos that are public and others that are private.

I'm looking to retire that repo soon, and replace it with a better one that has been thought out in advance and doesn't make as many compromises, does not take as many unnecessary risks, etc. But as a straw man to help you understand how to manage secrets in a different way that doesn't have that issue, maybe this will help!

[2fst4u]

I'm gonna be honest that leaves me far more confused than when I started. Flux is getting to be much more difficult to manage beyond the initial getting started docs.

I really wish the docs just had straightforward instructions for both of these parts, with genuine "type this command but put your values in here". The way the docs are currently is just too advanced for me.

I've already broken my cluster once, I just don't want to risk breaking it again. Thank you for your help but I just don't think I can make this happen without significant risk.

[kingdonb]

@2fst4u There is an intended goal of Weave GitOps, broadly, to make this all simpler and more "single entry point" with one easy way

I'm not sure there's anything in WG today which addresses your issue specifically, how to manage image update automation with the encrypted secrets, but that's a good target for some more docs. Thank you for the feedback and sorry for your bad experience.

[2fst4u]

No animosity on my end, just apprehension and a knowledge that I don't know enough. Genuinely thank you for the information so far.

[melkishengue]

@kingdonb I am experiencing the same issue from time to time on my cluster. Here is the output of the flux check command:

► checking prerequisites
✗ flux 0.36.0 <2.4.0 (new version is available, please upgrade)
✔ Kubernetes 1.30.5-gke.1699000 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v1.1.0
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.39.0
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.33.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.4.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.4.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.4.1
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta1
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1beta2
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta1
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta1
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta1
✔ receivers.notification.toolkit.fluxcd.io/v1beta1
✔ all checks passed

[kingdonb]

If you are running Flux 0.36.0 then you need to upgrade before we can offer any support or suggestions. 0.36.0 is well over a year old, and as stated in the previous reply the libgit2 dependency was removed in the next release, so this component is completely changed now.

If you still have issues on the latest release, it's only possible for the community to offer support when you provide a complete example - ideally this comes in the form of a public reproduction in a public repository, with the manifests set up as close as possible to the example that you followed.

[melkishengue]

I am running the latest fluxcd version, as you can see on the updated output (my flux cli version was old and I had to update it to the latest to get the correct output).

► checking prerequisites
✔ Kubernetes 1.30.5-gke.1699000 >=1.28.0-0
► checking version in cluster
✔ distribution: flux-v2.4.0
✔ bootstrapped: true
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v1.1.0
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.39.0
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.33.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.4.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.4.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.4.1
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta3
✔ buckets.source.toolkit.fluxcd.io/v1
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1
✔ helmreleases.helm.toolkit.fluxcd.io/v2
✔ helmrepositories.source.toolkit.fluxcd.io/v1
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta2
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta2
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta2
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta3
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed