[Airflow][v1.10.15] Scheduler no longer schedules after database restart

[bionic69]

ℹ️ This is my first message in GitHub Discussions

Description

👋 Hello everyone,
We encounter a problem with the scheduler when the database is restarted or upgraded.
The dags are sometimes no longer scheduled after the restart of the database until the scheduler is killed manually. 😢

Actual behavior

1. Database is shutting down
2. Scheduler retry connect to the database with the following error

FATAL:  the database system is shutting down

3. The database is down
4. Scheduler retry connect to the database with the following error

TCP/IP connections on port 5432?
Is the server running on host "$POSTGRES_URI" (XX.XX.XX.XX) and accepting
psycopg2.OperationalError: could not connect to server: Connection refused

6. The database is restarted
7. The scheduler no longer schedule dags 🔥
   7.1. At this step on, I have no more logs from the scheduler except Trimming airflow logs to 15 days.
   7.2. The pod scheduler state is running and we have no logs and no errors in k8s. 🤷‍♂️

Expected behavior

1. Database is shutting down
2. Scheduler retry connect to the database with the following error:

FATAL:  the database system is shutting down

4. The database is down
5. Scheduler retry connect to the database with the following error:

TCP/IP connections on port 5432?
Is the server running on host "$POSTGRES_URI" (XX.XX.XX.XX) and accepting
psycopg2.OperationalError: could not connect to server: Connection refused

6. The database is restarted
7. The scheduler reconnects to the database and the dags are scheduled 👍

Environment

• Airflow v1.10.15
• Deployed in kubernetes 1.22 using helm chart v1.0.0
• PostgreSQL 12.11

Can you help me understand why this sometimes happens? (The most important is step 7)

Thank you for your help

[potiuk]

I think you are basically in this situation now:

Airflow 1.10.15 is end of life (means not at all supported by the community) for almost 1.5 years now. And unless there are some other users who might helpy you - noone will. You deliberatly chose to stick to the old , unsupported version of the software so the price for it to pay is that you will not much of a help now.

My best advice is - migrate to latest version of Airlfow 2.

[bionic69]

That is what I thought !!
A big thank you for your reply 😉