Detected recursive loop when walking DAG directory /opt/airflow/dags: /opt/airflow/dags/978f1a596fd2f9a39c2c39fc626b21405d86b241 has appeared more than once

[jversolatocreditas]

Hello guys, I'm with a problem, when a using the chart with gitsync, my scheduler appear de error below:

Detected recursive loop when walking DAG directory /opt/airflow/dags: /opt/airflow/dags/978f1a596fd2f9a39c2c39fc626b21405d86b241 has appeared more than once

I accessed the /opt/airflow/dags and i found a symlink with the name repo, does anyone know how i can solve this problem?

[thesuperzapper]

@jversolatocreditas does your git repo itself contain a symbolic file link that includes something like ../.. or ../dags?

Because you might accidentally be pointing to the repo's own folder, causing a recursion.

NOTE: there don't get confused with the symbolic link from git-sync itself, which uses a link at /opt/airflow/dags/repo to point to the current synced version, for example, /opt/airflow/dags/978f1a596fd2f9a39c2c39fc626b21405d86b241.

[jversolatocreditas]

Hello, i resolved change the dags_folder = /opt/airflow/dags/repo

[thesuperzapper]

@jversolatocreditas are you saying your problem is fixed?

If so, what did you do?

[jversolatocreditas]

Yes, to resolve the problem, it's necessary change the config option dags_folder and put the value /opt/airflow/dags/repo

[omeryosef]

Thanks! solved my problem as well. it took time for me to understand that the /opt/airflow/dags/repo/ is the correct path for the dags. After changing the dags_folder as you mentioned, it solved my problem.

[arundeep78]

@jversolatocreditas : Do you mean you manually changed airflow.cfg file or did you add some configuration in values.yaml for this chart? I have airflow.cfg with dags_folder = /opt/airflow/dags while env variable in the same pod is AIRFLOW__CORE__DAGS_FOLDER=/opt/airflow/dags/repo/ . This is for 2.3.4 airflow version.

[mehtab123]

We are facing the similar issue, when we are upgrading the airflow to 2.5.1 version

[arundeep78]

In kubernetes ,it worked for me with chart version 8.6.1 and airflow tag: 2.4.3-python3.10. One mistake I was made was not executing helm dependency update before executing helm upgrade. As I had a custom chart which had airflow helm chart as dependency. And I think this helm chart still does not support airflow 2.5.1 officially. airflow version support

[mehtab123]

Thanks @arundeep78, This error is not coming in airflow 2.4.3 with python 3.10.

[arundeep78]

I would assume then that is due to the fact that this helm chart does not support airflow 2.5.x as in the reference link I shared above. But, that's just a guess.

[lucasfonsecads]

I go through the same error even exporting the local file to the airflow configuration variable airflow.cfg

export AIRFLOW_HOME=$PWD

[thesuperzapper]

@lucasfonsecads I am confused, why are you setting airflow.cfg or AIRFLOW_HOME?

The chart does not allow for you to use a airflow.cfg file, you must specify configs with the airflow.config value as described in the docs.

[arundeep78]

I would assume then that is due to the fact that this helm chart does not support airflow 2.5.x as in the reference link I shared above. But, that's just a guess. On Apr 5, 2023, at 15:26, mehtab123 ***@***.******@***.***>> wrote: Thanks @arundeep78<https://github.com/arundeep78>, This error is not coming in airflow 2.4.3 with python 3.10. — Reply to this email directly, view it on GitHub<#692 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AF3JUVEUR7HCKG6FGYEGXQDW7VXJBANCNFSM6AAAAAAURQDVKE>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[thesuperzapper]

@arundeep78 the chart is working fine with Airflow 2.5 in my testing, and I am planning to release version 8.7.0 of the chart with 2.5 as the default later today. So if you are having problems with Airflow 2.5, please tell me so I can resolve them.

[arundeep78]

@thesuperzapper To confirm that I tested upgrade of 8.6.1 chart with airflow 2.4.3-python-3.10 to chart 8.7.0 airflow 2.5.3-python-3.10 and it is working fine the dags gets scheduled and executed without issues. Btw, many thanks for this chart.

[mehtab123]

Hi @thesuperzapper we have the issue with airflow 2.5.3 python 3.10, could you please help us to upgrade the airflow 2.3.0 to 2.5.3.

Thanks