ArgoCD upgrade from Helm Chart 6.7.3 to 7.3.11 fails due to argo-redis secret job

[bhavinkotak]

Describe the bug

We have deployed ArgoCD in AKS Cluster 1.27. Redis-HA mode is enabled from argo-cd helm chart.
Argo is getting self managed. So installation/upgrade is done by updating ArgoCD app-set.yaml and not via helm upgrade command.

We tried upgrading ArgoCD using helm chart from v6.7.3 to v7.3.11. It tries to upgrade itself; however, there is a job (Name: argocd-redis-secret-init; Kind: Job) which just hangs.

argocd-redis-secret-init-xxxxx POD logs -

Checking for initial Redis password in secret argocd/argocd-redis at key auth.
Argo CD Redis secret state confirmed: secret name argocd-redis.
Password secret is configured properly.

We are able to see new secret getting created. However, no other error message or information gets posted.
We did restart all the deployment and sts resources for argocd; however no changes are observed.

Related helm chart

argo-cd

Helm chart version

7.3.11

To Reproduce

1. Install ArgoCD helm chart v6.7.3 - everything is up and running
2. Upgrade ArgoCD to helm chart v7.3.11 - it tries to upgrade itself; however there is a job (Name: argocd-redis-secret-init; Kind: Job) which just hangs.

argocd-redis-secret-init-xxxxx POD logs -

Checking for initial Redis password in secret argocd/argocd-redis at key auth.
Argo CD Redis secret state confirmed: secret name argocd-redis.
Password secret is configured properly.

We are able to see new secret getting created. However, no other error message or information gets posted.
We did restart all the deployment and sts resources for argocd; however no changes are observed.

Expected behavior

Successful upgrade of ArgoCD using latest helm chart 7.3.11

Screenshots

No response

Additional context

No response

[AssenDimitrov]

Same, you got any workaround? I am using argocd in GKE, without network policies...

[bhavinkotak]

nah.. I am still stuck on this issue...

[jkleinlercher]

Does the job hang in PreSync hook in ArgoCD UI? I have the same problem posted in argoproj/argo-cd#6880 (comment)

[jkleinlercher]

I think a ttl for the redis job could help

[yu-croco]

Maybe fixed in #2861 ? Please feel free to reopen if situation is not fixed yet.

[speedythesnail]

This issue should be re-opened. I am unable to do a fresh install of ArgoCD using the helm chart. I am able to do so using the operator in OpenShift, but in my Kind cluster it fails using the chart.

[keithdwilliams]

Facing the same issue, this needs to be re-opened.

[lboclboc]

Facing the same issue with the helm chart version 7.5.2. The job for creating the argocd-redis secret is not run at so several containers is complaining about the missing secret.

[DoumLaberge]

I have a similar issue when I try ton install the Argocd Helm Chart version 7.5.1. I've try the 6.11.1 version and the problem is al;ready there. I've the redis HA mode. I have this error:

Error: UPGRADE FAILED: pre-upgrade hooks failed: warning: Hook pre-upgrade argo-cd/templates/redis-secret-init/job.yaml failed: Job in version "v1" cannot be handled as a
Job: json: cannot unmarshal string into Go struct field PodSpec.spec.template.spec.imagePullSecrets of type []v1.LocalObjectReference

when I run it in debug mode, I can see there somthing not right in the argo-cd/templates/redis-secret-init/job.yaml
There are a white line between the key iomagePullSecrets and the value

# Source: argo-cd/templates/redis-secret-init/job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: argocd-redis-secret-init
  namespace: "argocd"
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-delete-policy": before-hook-creation
  labels:
    helm.sh/chart: argo-cd-7.5.2
    app.kubernetes.io/name: argocd-redis-secret-init
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/component: redis-secret-init
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: argocd
    app.kubernetes.io/version: "v2.12.3"
spec:
  ttlSecondsAfterFinished: 60
  template:
    metadata:
      labels:
        helm.sh/chart: argo-cd-7.5.2
        app.kubernetes.io/name: argocd-redis-secret-init
        app.kubernetes.io/instance: argocd
        app.kubernetes.io/component: redis-secret-init
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/part-of: argocd
        app.kubernetes.io/version: "v2.12.3"
    spec:
      imagePullSecrets:
        
        nexussecret
      containers:
      - command:
          - argocd
          - admin
          - redis-initial-password
        image: quay.io/argoproj/argocd:v2.12.3
        imagePullPolicy: IfNotPresent
        name: secret-init
        resources:
          {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
      restartPolicy: OnFailure
      serviceAccountName: argocd-redis-secret-init

[DoumLaberge]

The template redis-secret-init/job.yaml was introduce in 6.10.0 and doesn't seem to work since

[DoumLaberge]

I just find the error

{{- with .Values.global.imagePullSecrets }}
imagePullSecrets:
{{ toYaml . | nindent 8 }}
{{- end }}

There a missing '-' in front of the toYaml in the redis-secret-init/job.yaml