Flux2 upgrade itself from HelmRelease

[mo-saeed]

Hi,

I have flux2 helmrelease as a part of flux operations and it always fails when it tries to upgrade or to reconcile itself with this error

Helm upgrade failed: another operation (install/upgrade/rollback) is in progress

maybe i am doing sth wrong here, the helm release has this block

spec:
  releaseName: flux2
  upgrade:
    remediation:
      retries: -1
  install:
    remediation:
      retries: -1
  interval: 5m

would appreciate any advice.

[stefanprodan]

There is no support for using Helm to self upgrade, you need use flux bootstrap and the Flux GH Action.

PS. The Flux Helm charts are not part of the official distribution, any flux2 chart related issues should go to https://github.com/fluxcd-community/helm-charts

[mo-saeed]

Thanks @stefanprodan for your answer.

In general how to overcome this issue

◎ waiting for HelmRelease reconciliation
✗ Helm upgrade failed: another operation (install/upgrade/rollback) is in progress

I saw it also happening with other releases than flux2 so let's keep flux2 upgrade itself away from now and I want to find out what exactly can cause this issue from flux2.

[stefanprodan]

Flux helm-controller can't upgrade itself, the Helm storage gets into a deadlock state as the process gets killed by the upgrade and it leaves the lock in place.

[mo-saeed]

I understand but now I am talking about other applications than flux itself

csi-secrets-store-provider-aws                66m   False   Helm upgrade failed: another operation (install/upgrade/rollback) is in progress

[stefanprodan]

It's the same issue, a deadlock in the Helm storage. You need to delete the Helm storage secret. This will be address by @hiddeco in the helm-controller refactoring, see fluxcd/helm-controller#149 (comment)

[mo-saeed]

Thanks, does that mean that if this will be addressed, helm-controller will be able to upgrade itself ? or this will never work ?

[mo-saeed]

it's also super strange that I have this problem only happening in AWS EKS cluster but azure AKS cluster i don't have it, I am talking about flux2 upgrading itself... I assume if it doesn't work then it shouldn't work everywhere, correct?

[stefanprodan]

Thanks, does that mean that if this will be addressed, helm-controller will be able to upgrade itself ?

Yes I hope so, but nevertheless you’ll still need to disable waiting for the Flux release.

[camaeel]

I think this could be possible if helm-controller could apply the changes before it's pod is replaced. So apply the changes from helm chart without waitingg for the result and set some terminationGracePeriod (https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) before it is terminated.
Also another option could be to to have 2 replicas and terminate first the one that is not "leader" - but this would require custom controller for self patching that would terminate instances in proper order.