Delay in Queued Dttm

[burun]

Apache Airflow version: 2.1.1
Environment: apache/airflow:2.1.1 docker image

Hi, we are experiencing an issue with our Apache Airflow cluster, which comprises approximately 600 DAGs, out of which around 300 are active. Some of these active DAGs are scheduled to run at five-minute intervals. However, recently we have observed a delay in the queuing of these 5-minute tasks, as well as other tasks with longer interval.

Under normal circumstances, if the execution date is, say, 6/17/2023 12:40:00 AM, we would expect the task to be queued seconds after 6/17/2023 12:45:00 AM. But, of late, we've been noticing a delay, and the task can be queued as late as 6/17/2023 12:50:00 AM.

Our initial suspicion was directed towards the scheduler's resources. However, upon monitoring, we found that the scheduler's CPU and memory utilization remained relatively low. Similarly, we observed ample resources at the worker level as well, with tasks being executed promptly after queued.

Given this situation, we are currently at a loss on how to further investigate this issue. We would highly appreciate any insights or guidance regarding potential starting points for our investigation. Thank you in advance.

[potiuk]

There are likely several 1000s of issues and improvements fixed since 2.1.1 (released 2 years ago) - many of them fixing performance and many other aspects including the fact that there was 2.1.4 released that contains ONLY bugfixes since 2.1.1 (and likely few tens of them).

I think you have two choices:

1. go through the release notes (and the 1000s of improvements and fixes) and try to find if there are similar issues and hope that upgrading to this version help

or

2. just upgrade to latest version hoping for the same

I am very serious. I think nobody will spend time analysing in detail the kind of issues you have for version that you even did not patch to the latest bugfix in the 2.1 line if there is a high probability that your problems come from very old version that literally hundreds of people spent time on improving and fixing problems.

Upgrade

[Tangruilin]

Hey，we meet the same problem in 2.6.3。
The detail is in discussion #43673
our task cost about 4h from scheduled to queued most

[potiuk]

Can you please both (as requested above) migrate to latest version (2.10.3) and try it there? There is very little point in loosing time on 2 year old version of software where there is a new version with literally 1000s of bugfixes since.

The best course of action for both of you is to upgrade and if you see similar problems, report back.