dag gitsync sidecar make pod cannot start #785

jackchuong · 2023-09-13T10:29:47Z

jackchuong
Sep 13, 2023

Hi all,
It works fine if I use pvc for dags, here helm chart values:

airflow:
  extraPipPackages:
    - "pyspark==3.4.1"
    - "apache-airflow-providers-apache-spark==2.1.3"
dags:
  path: /opt/airflow/dags

  persistence:
    enabled: true
    existingClaim: "dags-pvc" # pvc "dags-pvc" existed/created
    storageClass: "nfs-idcnas002"
    accessMode: ReadWriteMany
    size: 1Gi

  gitSync:
    enabled: false

However, If I want to use Git-Sync Sidecar SSH for dags, here helm chart values:

airflow:
  extraPipPackages:
    - "pyspark==3.4.1"
    - "apache-airflow-providers-apache-spark==2.1.3"
dags:
  path: /opt/airflow/dags

  persistence:
    enabled: false

  gitSync:
    enabled: true
    repo: "[email protected]/jun.dung/airflow-dags.git" # this repo existed, pushed test.py to this repo already
    repoSubPath: ""
    branch: main
    revision: HEAD
    depth: 1
    syncWait: 60
    syncTimeout: 20
    submodules: recursive
    sshSecret: "airflow-ssh-git-secret" # secret "airflow-ssh-git-secret" existed/created from file id_rsa and user with this id_rsa can clone/push from/to git repo successfully
    sshSecretKey: "id_rsa"
    sshKnownHosts: ""
    maxFailures: 2

kubectl -n ebis describe secret airflow-ssh-git-secret
Name:         airflow-ssh-git-secret
Namespace:    ebis
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
id_rsa:  2622 bytes

Some pods cannot be started

kubectl -n ebis get pod
NAME                                    READY   STATUS     RESTARTS       AGE
airflow-postgresql-0                    1/1     Running    0              10m
airflow-pgbouncer-5d47c57dcb-fk9gc      1/1     Running    0              10m
airflow-redis-master-0                  1/1     Running    0              10m
airflow-db-migrations-7748c5b7c-c95lr   0/2     Init:1/3   3 (116s ago)   10m
airflow-worker-0                        0/2     Init:1/4   3 (117s ago)   10m
airflow-triggerer-65bcd96dbd-8t62z      0/2     Init:1/4   3 (116s ago)   10m
airflow-sync-users-8b88b67d9-vqwbk      0/2     Init:1/4   3 (37s ago)    10m
airflow-web-69fb55787c-97r2n            0/2     Init:1/4   3 (38s ago)    10m
airflow-flower-696cf66dd4-b74xb         0/2     Init:1/4   3 (37s ago)    10m
airflow-scheduler-6b7bf5b894-stsxh      0/2     Init:1/4   3 (35s ago)    10m

kubectl -n ebis get pod
NAME                                    READY   STATUS                  RESTARTS        AGE
airflow-postgresql-0                    1/1     Running                 0               20m
airflow-pgbouncer-5d47c57dcb-fk9gc      1/1     Running                 0               20m
airflow-redis-master-0                  1/1     Running                 0               20m
airflow-sync-users-8b88b67d9-vqwbk      0/2     Init:CrashLoopBackOff   5 (2m47s ago)   20m
airflow-web-69fb55787c-97r2n            0/2     Init:CrashLoopBackOff   5 (2m44s ago)   20m
airflow-scheduler-6b7bf5b894-stsxh      0/2     Init:CrashLoopBackOff   5 (2m40s ago)   20m
airflow-flower-696cf66dd4-b74xb         0/2     Init:CrashLoopBackOff   5 (2m35s ago)   20m
airflow-worker-0                        0/2     Init:1/4                6 (4m3s ago)    20m
airflow-triggerer-65bcd96dbd-8t62z      0/2     Init:1/4                6 (4m ago)      20m
airflow-db-migrations-7748c5b7c-c95lr   0/2     Init:1/3                6 (3m59s ago)   20m

kubectl -n ebis logs -p airflow-db-migrations-7748c5b7c-c95lr
Defaulted container "db-migrations" out of: db-migrations, dags-git-sync, install-pip-packages (init), dags-git-clone (init), check-db (init)
Error from server (BadRequest): previous terminated container "db-migrations" in pod "airflow-db-migrations-7748c5b7c-c95lr" not found

kubectl -n ebis describe pod airflow-db-migrations-7748c5b7c-c95lr
Name:             airflow-db-migrations-7748c5b7c-c95lr
Namespace:        ebis
Priority:         0
Service Account:  airflow
Node:             k3s-dc-master1/192.168.0.200
Start Time:       Wed, 13 Sep 2023 17:09:53 +0700
Labels:           app=airflow
                  component=db-migrations
                  pod-template-hash=7748c5b7c
                  release=airflow
Annotations:      checksum/db-migrations-script: 37898f38b90abd06081105d992362ec9e0d0015123b69e758e59031a9e6ddfc9
                  checksum/secret-config-envs: fbcfbef63b078e09a51047589ad82590018af8901d43b729891d52511a017de4
                  checksum/secret-local-settings: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                  cluster-autoscaler.kubernetes.io/safe-to-evict: true
Status:           Pending
IP:               10.42.0.134
IPs:
  IP:           10.42.0.134
Controlled By:  ReplicaSet/airflow-db-migrations-7748c5b7c
Init Containers:
  install-pip-packages:
    Container ID:  containerd://37230afdce5c7637cedda05c06ecd2d37ef8c7d92049fe40ed13ae11235f47a3
    Image:         apache/airflow:2.5.3-python3.8
    Image ID:      docker.io/apache/airflow@sha256:92625ecda829d51b9a7ae35bf32cec8e75b1784098165d7d95658cfbbbb02700
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/dumb-init
      --
      /entrypoint
    Args:
      bash
      -c
      unset PYTHONUSERBASE && \
      pip freeze | grep -i -e "apache-airflow=="  > protected-packages.txt && \
      pip install --constraint ./protected-packages.txt --user "pyspark==3.4.1" "apache-airflow-providers-apache-spark==2.1.3"  && \
      echo "copying '/home/airflow/.local/' to '/opt/home-airflow-local/.local/'..." && \
      rsync -ah --stats --delete /home/airflow/.local/ /opt/home-airflow-local/.local/

    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 13 Sep 2023 17:09:54 +0700
      Finished:     Wed, 13 Sep 2023 17:11:47 +0700
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      airflow-config-envs  Secret  Optional: false
    Environment:
      DATABASE_USER:               postgres
      DATABASE_PASSWORD:           <set to the key 'postgresql-password' in secret 'airflow-postgresql'>  Optional: false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'airflow-redis'>            Optional: false
      CONNECTION_CHECK_MAX_COUNT:  0
    Mounts:
      /opt/home-airflow-local from home-airflow-local (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bkm9c (ro)
  dags-git-clone:
    Container ID:   containerd://9bff7f1188282e072a9b280d3e63332f63a33d02be66915cf55f02c630a6d593
    Image:          registry.k8s.io/git-sync/git-sync:v3.6.5
    Image ID:       registry.k8s.io/git-sync/git-sync@sha256:7231f6c2284758b91caed71e4e596413df31ac4467de9b596dc6b386b82f624f
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 13 Sep 2023 17:21:23 +0700
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 13 Sep 2023 17:18:32 +0700
      Finished:     Wed, 13 Sep 2023 17:20:32 +0700
    Ready:          False
    Restart Count:  4
    Environment Variables from:
      airflow-config-envs  Secret  Optional: false
    Environment:
      GIT_SYNC_ONE_TIME:           true
      GIT_SYNC_ROOT:               /dags
      GIT_SYNC_DEST:               repo
      GIT_SYNC_REPO:               [email protected]/jun.dung/airflow-dags.git
      GIT_SYNC_BRANCH:             main
      GIT_SYNC_REV:                HEAD
      GIT_SYNC_DEPTH:              1
      GIT_SYNC_WAIT:               60
      GIT_SYNC_TIMEOUT:            20
      GIT_SYNC_ADD_USER:           true
      GIT_SYNC_MAX_SYNC_FAILURES:  2
      GIT_SYNC_SUBMODULES:         recursive
      GIT_SYNC_SSH:                true
      GIT_SSH_KEY_FILE:            /etc/git-secret/id_rsa
      GIT_KNOWN_HOSTS:             false
      DATABASE_USER:               postgres
      DATABASE_PASSWORD:           <set to the key 'postgresql-password' in secret 'airflow-postgresql'>  Optional: false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'airflow-redis'>            Optional: false
      CONNECTION_CHECK_MAX_COUNT:  0
    Mounts:
      /dags from dags-data (rw)
      /etc/git-secret/id_rsa from git-secret (ro,path="id_rsa")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bkm9c (ro)
  check-db:
    Container ID:
    Image:         apache/airflow:2.5.3-python3.8
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/dumb-init
      --
      /entrypoint
    Args:
      bash
      -c
      exec timeout 60s airflow db check
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      airflow-config-envs  Secret  Optional: false
    Environment:
      DATABASE_USER:               postgres
      DATABASE_PASSWORD:           <set to the key 'postgresql-password' in secret 'airflow-postgresql'>  Optional: false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'airflow-redis'>            Optional: false
      CONNECTION_CHECK_MAX_COUNT:  0
    Mounts:
      /home/airflow/.local from home-airflow-local (rw,path=".local")
      /opt/airflow/dags from dags-data (rw)
      /opt/airflow/logs from logs-data (rw,path="airflow-logs")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bkm9c (ro)
Containers:
  db-migrations:
    Container ID:
    Image:         apache/airflow:2.5.3-python3.8
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/dumb-init
      --
      /entrypoint
    Args:
      python
      -u
      /mnt/scripts/db_migrations.py
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      airflow-config-envs  Secret  Optional: false
    Environment:
      DATABASE_USER:               postgres
      DATABASE_PASSWORD:           <set to the key 'postgresql-password' in secret 'airflow-postgresql'>  Optional: false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'airflow-redis'>            Optional: false
      CONNECTION_CHECK_MAX_COUNT:  0
    Mounts:
      /home/airflow/.local from home-airflow-local (rw,path=".local")
      /mnt/scripts from scripts (ro)
      /opt/airflow/dags from dags-data (rw)
      /opt/airflow/logs from logs-data (rw,path="airflow-logs")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bkm9c (ro)
  dags-git-sync:
    Container ID:
    Image:          registry.k8s.io/git-sync/git-sync:v3.6.5
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      airflow-config-envs  Secret  Optional: false
    Environment:
      GIT_SYNC_ROOT:               /dags
      GIT_SYNC_DEST:               repo
      GIT_SYNC_REPO:               [email protected]/jun.dung/airflow-dags.git
      GIT_SYNC_BRANCH:             main
      GIT_SYNC_REV:                HEAD
      GIT_SYNC_DEPTH:              1
      GIT_SYNC_WAIT:               60
      GIT_SYNC_TIMEOUT:            20
      GIT_SYNC_ADD_USER:           true
      GIT_SYNC_MAX_SYNC_FAILURES:  2
      GIT_SYNC_SUBMODULES:         recursive
      GIT_SYNC_SSH:                true
      GIT_SSH_KEY_FILE:            /etc/git-secret/id_rsa
      GIT_KNOWN_HOSTS:             false
      DATABASE_USER:               postgres
      DATABASE_PASSWORD:           <set to the key 'postgresql-password' in secret 'airflow-postgresql'>  Optional: false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'airflow-redis'>            Optional: false
      CONNECTION_CHECK_MAX_COUNT:  0
    Mounts:
      /dags from dags-data (rw)
      /etc/git-secret/id_rsa from git-secret (ro,path="id_rsa")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bkm9c (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  dags-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  logs-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  ebis-pvc
    ReadOnly:   false
  git-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  airflow-ssh-git-secret
    Optional:    false
  home-airflow-local:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  scripts:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  airflow-db-migrations
    Optional:    false
  kube-api-access-bkm9c:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  12m                  default-scheduler  Successfully assigned ebis/airflow-db-migrations-7748c5b7c-c95lr to k3s-dc-master1
  Normal   Pulled     12m                  kubelet            Container image "apache/airflow:2.5.3-python3.8" already present on machine
  Normal   Created    12m                  kubelet            Created container install-pip-packages
  Normal   Started    12m                  kubelet            Started container install-pip-packages
  Warning  BackOff    55s (x7 over 6m11s)  kubelet            Back-off restarting failed container dags-git-clone in pod airflow-db-migrations-7748c5b7c-c95lr_ebis(f5211192-be59-4fb2-9ae1-d89c11078a09)
  Normal   Pulled     43s (x5 over 10m)    kubelet            Container image "registry.k8s.io/git-sync/git-sync:v3.6.5" already present on machine
  Normal   Created    43s (x5 over 10m)    kubelet            Created container dags-git-clone
  Normal   Started    43s (x5 over 10m)    kubelet            Started container dags-git-clone

kubectl -n ebis logs -p airflow-worker-0
Defaulted container "airflow-worker" out of: airflow-worker, dags-git-sync, install-pip-packages (init), dags-git-clone (init), check-db (init), wait-for-db-migrations (init)
Error from server (BadRequest): previous terminated container "airflow-worker" in pod "airflow-worker-0" not found

kubectl -n ebis describe pod airflow-worker-0
Name:             airflow-worker-0
Namespace:        ebis
Priority:         0
Service Account:  airflow
Node:             k3s-dc-master1/192.168.0.200
Start Time:       Wed, 13 Sep 2023 17:09:53 +0700
Labels:           app=airflow
                  component=worker
                  controller-revision-hash=airflow-worker-6dcbc88d48
                  release=airflow
                  statefulset.kubernetes.io/pod-name=airflow-worker-0
Annotations:      checksum/secret-config-envs: fbcfbef63b078e09a51047589ad82590018af8901d43b729891d52511a017de4
                  checksum/secret-local-settings: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                  cluster-autoscaler.kubernetes.io/safe-to-evict: true
Status:           Pending
IP:               10.42.0.135
IPs:
  IP:           10.42.0.135
Controlled By:  StatefulSet/airflow-worker
Init Containers:
  install-pip-packages:
    Container ID:  containerd://4aa5d8c741cdfe07ff255c8c10c73a7711dbf83f6815e76f561d84b233e4d800
    Image:         apache/airflow:2.5.3-python3.8
    Image ID:      docker.io/apache/airflow@sha256:92625ecda829d51b9a7ae35bf32cec8e75b1784098165d7d95658cfbbbb02700
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/dumb-init
      --
      /entrypoint
    Args:
      bash
      -c
      unset PYTHONUSERBASE && \
      pip freeze | grep -i -e "apache-airflow=="  > protected-packages.txt && \
      pip install --constraint ./protected-packages.txt --user "pyspark==3.4.1" "apache-airflow-providers-apache-spark==2.1.3"  && \
      echo "copying '/home/airflow/.local/' to '/opt/home-airflow-local/.local/'..." && \
      rsync -ah --stats --delete /home/airflow/.local/ /opt/home-airflow-local/.local/

    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 13 Sep 2023 17:09:54 +0700
      Finished:     Wed, 13 Sep 2023 17:11:36 +0700
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      airflow-config-envs  Secret  Optional: false
    Environment:
      DATABASE_USER:               postgres
      DATABASE_PASSWORD:           <set to the key 'postgresql-password' in secret 'airflow-postgresql'>  Optional: false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'airflow-redis'>            Optional: false
      CONNECTION_CHECK_MAX_COUNT:  0
    Mounts:
      /opt/home-airflow-local from home-airflow-local (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fzrbr (ro)
  dags-git-clone:
    Container ID:   containerd://6c76eb90d7a9b3fa0257451e1e8a34b9789189faf3c4c64ee8111f1e6295059e
    Image:          registry.k8s.io/git-sync/git-sync:v3.6.5
    Image ID:       registry.k8s.io/git-sync/git-sync@sha256:7231f6c2284758b91caed71e4e596413df31ac4467de9b596dc6b386b82f624f
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 13 Sep 2023 17:24:47 +0700
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 13 Sep 2023 17:21:17 +0700
      Finished:     Wed, 13 Sep 2023 17:23:17 +0700
    Ready:          False
    Restart Count:  5
    Environment Variables from:
      airflow-config-envs  Secret  Optional: false
    Environment:
      GIT_SYNC_ONE_TIME:           true
      GIT_SYNC_ROOT:               /dags
      GIT_SYNC_DEST:               repo
      GIT_SYNC_REPO:               [email protected]/jun.dung/airflow-dags.git
      GIT_SYNC_BRANCH:             main
      GIT_SYNC_REV:                HEAD
      GIT_SYNC_DEPTH:              1
      GIT_SYNC_WAIT:               60
      GIT_SYNC_TIMEOUT:            20
      GIT_SYNC_ADD_USER:           true
      GIT_SYNC_MAX_SYNC_FAILURES:  2
      GIT_SYNC_SUBMODULES:         recursive
      GIT_SYNC_SSH:                true
      GIT_SSH_KEY_FILE:            /etc/git-secret/id_rsa
      GIT_KNOWN_HOSTS:             false
      DATABASE_USER:               postgres
      DATABASE_PASSWORD:           <set to the key 'postgresql-password' in secret 'airflow-postgresql'>  Optional: false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'airflow-redis'>            Optional: false
      CONNECTION_CHECK_MAX_COUNT:  0
    Mounts:
      /dags from dags-data (rw)
      /etc/git-secret/id_rsa from git-secret (ro,path="id_rsa")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fzrbr (ro)
  check-db:
    Container ID:
    Image:         apache/airflow:2.5.3-python3.8
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/dumb-init
      --
      /entrypoint
    Args:
      bash
      -c
      exec timeout 60s airflow db check
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      airflow-config-envs  Secret  Optional: false
    Environment:
      DATABASE_USER:               postgres
      DATABASE_PASSWORD:           <set to the key 'postgresql-password' in secret 'airflow-postgresql'>  Optional: false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'airflow-redis'>            Optional: false
      CONNECTION_CHECK_MAX_COUNT:  0
    Mounts:
      /home/airflow/.local from home-airflow-local (rw,path=".local")
      /opt/airflow/dags from dags-data (rw)
      /opt/airflow/logs from logs-data (rw,path="airflow-logs")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fzrbr (ro)
  wait-for-db-migrations:
    Container ID:
    Image:         apache/airflow:2.5.3-python3.8
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/dumb-init
      --
      /entrypoint
    Args:
      bash
      -c
      exec airflow db check-migrations -t 60
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      airflow-config-envs  Secret  Optional: false
    Environment:
      DATABASE_USER:               postgres
      DATABASE_PASSWORD:           <set to the key 'postgresql-password' in secret 'airflow-postgresql'>  Optional: false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'airflow-redis'>            Optional: false
      CONNECTION_CHECK_MAX_COUNT:  0
    Mounts:
      /home/airflow/.local from home-airflow-local (rw,path=".local")
      /opt/airflow/dags from dags-data (rw)
      /opt/airflow/logs from logs-data (rw,path="airflow-logs")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fzrbr (ro)
Containers:
  airflow-worker:
    Container ID:
    Image:         apache/airflow:2.5.3-python3.8
    Image ID:
    Port:          8793/TCP
    Host Port:     0/TCP
    Command:
      /usr/bin/dumb-init
      --
      /entrypoint
    Args:
      bash
      -c
      exec airflow celery worker
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  2Gi
    Requests:
      cpu:     500m
      memory:  2Gi
    Liveness:  exec [/usr/bin/dumb-init -- /entrypoint python -Wignore -c import os
import sys
import subprocess
from celery import Celery
from celery.app.control import Inspect
from typing import List

def run_command(cmd: List[str]) -> str:
    process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
    output, error = process.communicate()
    if error is not None:
        raise Exception(error)
    else:
        return output.decode(encoding="utf-8")

broker_url = run_command(["bash", "-c", "eval $AIRFLOW__CELERY__BROKER_URL_CMD"])
local_celery_host = f"celery@{os.environ['HOSTNAME']}"
app = Celery(broker=broker_url)

# ping the local celery worker to see if it's ok
i = Inspect(app=app, destination=[local_celery_host], timeout=5.0)
ping_responses = i.ping()
if local_celery_host not in ping_responses:
    sys.exit(f"celery worker '{local_celery_host}' did not respond to ping")
] delay=10s timeout=60s period=30s #success=1 #failure=5
    Environment Variables from:
      airflow-config-envs  Secret  Optional: false
    Environment:
      DATABASE_USER:               postgres
      DATABASE_PASSWORD:           <set to the key 'postgresql-password' in secret 'airflow-postgresql'>  Optional: false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'airflow-redis'>            Optional: false
      CONNECTION_CHECK_MAX_COUNT:  0
      DUMB_INIT_SETSID:            0
    Mounts:
      /home/airflow/.local from home-airflow-local (rw,path=".local")
      /opt/airflow/dags from dags-data (rw)
      /opt/airflow/logs from logs-data (rw,path="airflow-logs")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fzrbr (ro)
  dags-git-sync:
    Container ID:
    Image:          registry.k8s.io/git-sync/git-sync:v3.6.5
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      airflow-config-envs  Secret  Optional: false
    Environment:
      GIT_SYNC_ROOT:               /dags
      GIT_SYNC_DEST:               repo
      GIT_SYNC_REPO:               [email protected]/jun.dung/airflow-dags.git
      GIT_SYNC_BRANCH:             main
      GIT_SYNC_REV:                HEAD
      GIT_SYNC_DEPTH:              1
      GIT_SYNC_WAIT:               60
      GIT_SYNC_TIMEOUT:            20
      GIT_SYNC_ADD_USER:           true
      GIT_SYNC_MAX_SYNC_FAILURES:  2
      GIT_SYNC_SUBMODULES:         recursive
      GIT_SYNC_SSH:                true
      GIT_SSH_KEY_FILE:            /etc/git-secret/id_rsa
      GIT_KNOWN_HOSTS:             false
      DATABASE_USER:               postgres
      DATABASE_PASSWORD:           <set to the key 'postgresql-password' in secret 'airflow-postgresql'>  Optional: false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'airflow-redis'>            Optional: false
      CONNECTION_CHECK_MAX_COUNT:  0
    Mounts:
      /dags from dags-data (rw)
      /etc/git-secret/id_rsa from git-secret (ro,path="id_rsa")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fzrbr (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  dags-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  logs-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  ebis-pvc
    ReadOnly:   false
  git-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  airflow-ssh-git-secret
    Optional:    false
  home-airflow-local:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-fzrbr:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  14m                  default-scheduler  Successfully assigned ebis/airflow-worker-0 to k3s-dc-master1
  Normal   Pulled     14m                  kubelet            Container image "apache/airflow:2.5.3-python3.8" already present on machine
  Normal   Created    14m                  kubelet            Created container install-pip-packages
  Normal   Started    14m                  kubelet            Started container install-pip-packages
  Normal   Pulled     3m34s (x5 over 13m)  kubelet            Container image "registry.k8s.io/git-sync/git-sync:v3.6.5" already present on machine
  Normal   Created    3m34s (x5 over 13m)  kubelet            Created container dags-git-clone
  Normal   Started    3m34s (x5 over 13m)  kubelet            Started container dags-git-clone
  Warning  BackOff    64s (x9 over 9m1s)   kubelet            Back-off restarting failed container dags-git-clone in pod airflow-worker-0_ebis(0c627b51-6d01-4418-bca8-3bd57d07a83a)

kubectl -n ebis logs -p airflow-sync-users-8b88b67d9-vqwbk
Defaulted container "sync-airflow-users" out of: sync-airflow-users, dags-git-sync, install-pip-packages (init), dags-git-clone (init), check-db (init), wait-for-db-migrations (init)
Error from server (BadRequest): previous terminated container "sync-airflow-users" in pod "airflow-sync-users-8b88b67d9-vqwbk" not found

kubectl -n ebis describe pod airflow-sync-users-8b88b67d9-vqwbk
Name:             airflow-sync-users-8b88b67d9-vqwbk
Namespace:        ebis
Priority:         0
Service Account:  airflow
Node:             k3s-dc-worker1/192.168.0.201
Start Time:       Wed, 13 Sep 2023 17:09:53 +0700
Labels:           app=airflow
                  component=sync-users
                  pod-template-hash=8b88b67d9
                  release=airflow
Annotations:      checksum/secret-config-envs: fbcfbef63b078e09a51047589ad82590018af8901d43b729891d52511a017de4
                  checksum/secret-local-settings: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
                  checksum/sync-users-script: 3492f833120e4914ee2baffc208eaf928ffd6fb840ab3b60f94258d5146f1d7f
                  cluster-autoscaler.kubernetes.io/safe-to-evict: true
Status:           Pending
IP:               10.42.1.43
IPs:
  IP:           10.42.1.43
Controlled By:  ReplicaSet/airflow-sync-users-8b88b67d9
Init Containers:
  install-pip-packages:
    Container ID:  containerd://b6d5b78b5c7b3bc79c2cb3d0c0f534f61fce904b05066c998e4d02ef0afed464
    Image:         apache/airflow:2.5.3-python3.8
    Image ID:      docker.io/apache/airflow@sha256:92625ecda829d51b9a7ae35bf32cec8e75b1784098165d7d95658cfbbbb02700
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/dumb-init
      --
      /entrypoint
    Args:
      bash
      -c
      unset PYTHONUSERBASE && \
      pip freeze | grep -i -e "apache-airflow=="  > protected-packages.txt && \
      pip install --constraint ./protected-packages.txt --user "pyspark==3.4.1" "apache-airflow-providers-apache-spark==2.1.3"  && \
      echo "copying '/home/airflow/.local/' to '/opt/home-airflow-local/.local/'..." && \
      rsync -ah --stats --delete /home/airflow/.local/ /opt/home-airflow-local/.local/

    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 13 Sep 2023 17:09:55 +0700
      Finished:     Wed, 13 Sep 2023 17:12:26 +0700
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      airflow-config-envs  Secret  Optional: false
    Environment:
      DATABASE_USER:               postgres
      DATABASE_PASSWORD:           <set to the key 'postgresql-password' in secret 'airflow-postgresql'>  Optional: false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'airflow-redis'>            Optional: false
      CONNECTION_CHECK_MAX_COUNT:  0
    Mounts:
      /opt/home-airflow-local from home-airflow-local (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kxcmb (ro)
  dags-git-clone:
    Container ID:   containerd://428d1f86ed73d0b2ed8e54fc58cb0cd0d1701c90ce0d4da08826b6861d6143f5
    Image:          registry.k8s.io/git-sync/git-sync:v3.6.5
    Image ID:       registry.k8s.io/git-sync/git-sync@sha256:7231f6c2284758b91caed71e4e596413df31ac4467de9b596dc6b386b82f624f
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 13 Sep 2023 17:26:03 +0700
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 13 Sep 2023 17:22:34 +0700
      Finished:     Wed, 13 Sep 2023 17:24:34 +0700
    Ready:          False
    Restart Count:  5
    Environment Variables from:
      airflow-config-envs  Secret  Optional: false
    Environment:
      GIT_SYNC_ONE_TIME:           true
      GIT_SYNC_ROOT:               /dags
      GIT_SYNC_DEST:               repo
      GIT_SYNC_REPO:               [email protected]/jun.dung/airflow-dags.git
      GIT_SYNC_BRANCH:             main
      GIT_SYNC_REV:                HEAD
      GIT_SYNC_DEPTH:              1
      GIT_SYNC_WAIT:               60
      GIT_SYNC_TIMEOUT:            20
      GIT_SYNC_ADD_USER:           true
      GIT_SYNC_MAX_SYNC_FAILURES:  2
      GIT_SYNC_SUBMODULES:         recursive
      GIT_SYNC_SSH:                true
      GIT_SSH_KEY_FILE:            /etc/git-secret/id_rsa
      GIT_KNOWN_HOSTS:             false
      DATABASE_USER:               postgres
      DATABASE_PASSWORD:           <set to the key 'postgresql-password' in secret 'airflow-postgresql'>  Optional: false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'airflow-redis'>            Optional: false
      CONNECTION_CHECK_MAX_COUNT:  0
    Mounts:
      /dags from dags-data (rw)
      /etc/git-secret/id_rsa from git-secret (ro,path="id_rsa")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kxcmb (ro)
  check-db:
    Container ID:
    Image:         apache/airflow:2.5.3-python3.8
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/dumb-init
      --
      /entrypoint
    Args:
      bash
      -c
      exec timeout 60s airflow db check
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      airflow-config-envs  Secret  Optional: false
    Environment:
      DATABASE_USER:               postgres
      DATABASE_PASSWORD:           <set to the key 'postgresql-password' in secret 'airflow-postgresql'>  Optional: false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'airflow-redis'>            Optional: false
      CONNECTION_CHECK_MAX_COUNT:  0
    Mounts:
      /home/airflow/.local from home-airflow-local (rw,path=".local")
      /opt/airflow/dags from dags-data (rw)
      /opt/airflow/logs from logs-data (rw,path="airflow-logs")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kxcmb (ro)
  wait-for-db-migrations:
    Container ID:
    Image:         apache/airflow:2.5.3-python3.8
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/dumb-init
      --
      /entrypoint
    Args:
      bash
      -c
      exec airflow db check-migrations -t 60
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      airflow-config-envs  Secret  Optional: false
    Environment:
      DATABASE_USER:               postgres
      DATABASE_PASSWORD:           <set to the key 'postgresql-password' in secret 'airflow-postgresql'>  Optional: false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'airflow-redis'>            Optional: false
      CONNECTION_CHECK_MAX_COUNT:  0
    Mounts:
      /home/airflow/.local from home-airflow-local (rw,path=".local")
      /opt/airflow/dags from dags-data (rw)
      /opt/airflow/logs from logs-data (rw,path="airflow-logs")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kxcmb (ro)
Containers:
  sync-airflow-users:
    Container ID:
    Image:         apache/airflow:2.5.3-python3.8
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/dumb-init
      --
      /entrypoint
    Args:
      python
      -u
      /mnt/scripts/sync_users.py
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      airflow-config-envs  Secret  Optional: false
    Environment:
      DATABASE_USER:               postgres
      DATABASE_PASSWORD:           <set to the key 'postgresql-password' in secret 'airflow-postgresql'>  Optional: false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'airflow-redis'>            Optional: false
      CONNECTION_CHECK_MAX_COUNT:  0
    Mounts:
      /home/airflow/.local from home-airflow-local (rw,path=".local")
      /mnt/scripts from scripts (ro)
      /opt/airflow/dags from dags-data (rw)
      /opt/airflow/logs from logs-data (rw,path="airflow-logs")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kxcmb (ro)
  dags-git-sync:
    Container ID:
    Image:          registry.k8s.io/git-sync/git-sync:v3.6.5
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      airflow-config-envs  Secret  Optional: false
    Environment:
      GIT_SYNC_ROOT:               /dags
      GIT_SYNC_DEST:               repo
      GIT_SYNC_REPO:               [email protected]/jun.dung/airflow-dags.git
      GIT_SYNC_BRANCH:             main
      GIT_SYNC_REV:                HEAD
      GIT_SYNC_DEPTH:              1
      GIT_SYNC_WAIT:               60
      GIT_SYNC_TIMEOUT:            20
      GIT_SYNC_ADD_USER:           true
      GIT_SYNC_MAX_SYNC_FAILURES:  2
      GIT_SYNC_SUBMODULES:         recursive
      GIT_SYNC_SSH:                true
      GIT_SSH_KEY_FILE:            /etc/git-secret/id_rsa
      GIT_KNOWN_HOSTS:             false
      DATABASE_USER:               postgres
      DATABASE_PASSWORD:           <set to the key 'postgresql-password' in secret 'airflow-postgresql'>  Optional: false
      REDIS_PASSWORD:              <set to the key 'redis-password' in secret 'airflow-redis'>            Optional: false
      CONNECTION_CHECK_MAX_COUNT:  0
    Mounts:
      /dags from dags-data (rw)
      /etc/git-secret/id_rsa from git-secret (ro,path="id_rsa")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kxcmb (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  dags-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  logs-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  ebis-pvc
    ReadOnly:   false
  git-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  airflow-ssh-git-secret
    Optional:    false
  home-airflow-local:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  scripts:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  airflow-sync-users
    Optional:    false
  kube-api-access-kxcmb:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  17m                  default-scheduler  Successfully assigned ebis/airflow-sync-users-8b88b67d9-vqwbk to k3s-dc-worker1
  Normal   Pulled     17m                  kubelet            Container image "apache/airflow:2.5.3-python3.8" already present on machine
  Normal   Created    17m                  kubelet            Created container install-pip-packages
  Normal   Started    17m                  kubelet            Started container install-pip-packages
  Normal   Pulled     4m39s (x5 over 14m)  kubelet            Container image "registry.k8s.io/git-sync/git-sync:v3.6.5" already present on machine
  Normal   Created    4m39s (x5 over 14m)  kubelet            Created container dags-git-clone
  Normal   Started    4m39s (x5 over 14m)  kubelet            Started container dags-git-clone
  Warning  BackOff    118s (x10 over 10m)  kubelet            Back-off restarting failed container dags-git-clone in pod airflow-sync-users-8b88b67d9-vqwbk_ebis(695f6abf-182d-41b3-a546-8a3d0b832dbc)

Other pod's logs and descriptions are same.
Please give me some advice, thank you very much.

Answered by thesuperzapper

Sep 14, 2023

@jackchuong Usually, if the dags-git-clone container in the pods is failing, it's because of a git permission or network access error.

Check the actual logs of one of the dags-git-clone Pods, and it should reveal which one it is.

View full answer

thesuperzapper · 2023-09-14T19:35:12Z

thesuperzapper
Sep 14, 2023
Maintainer

@jackchuong Usually, if the dags-git-clone container in the pods is failing, it's because of a git permission or network access error.

Check the actual logs of one of the dags-git-clone Pods, and it should reveal which one it is.

0 replies

jackchuong · 2023-09-15T03:36:42Z

jackchuong
Sep 15, 2023
Author

@thesuperzapper Thank you very much , I figure out the issue : repo's url is wrong syntax , it works fine now.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dag gitsync sidecar make pod cannot start #785

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments

{{title}}

{{title}}

Select a reply

dag gitsync sidecar make pod cannot start #785

jackchuong Sep 13, 2023

Replies: 2 comments

thesuperzapper Sep 14, 2023 Maintainer

jackchuong Sep 15, 2023 Author

jackchuong
Sep 13, 2023

thesuperzapper
Sep 14, 2023
Maintainer

jackchuong
Sep 15, 2023
Author