Limit of 100 pipelines Imported from Airflow 2.7 (MySQL backend db) in OpenMetadata 1.5.5

[d-gorodnov]

Affected module
Ingestion Framework

Describe the bug
When ingesting Airflow metadata using OpenMetadata version 1.5.5, only the top 100 DAGs are imported, even though the serialized_dag table in the Airflow database contains more records. There appears to be no configuration parameter to adjust this limit.

To Reproduce

1. Start OpenMetadata using the docker-compose.yml file downloaded from GitHub.
2. Generate more than 100 DAG records in the serialized_dag table within the airflow_db.
3. Attempt to ingest internal Airflow metadata.
4. Observe that only 100 pipelines are displayed in the OpenMetadata UI.

Also you can try to ingest pipelines from random Airflow with backend db MySQL 8 and DAGs amount >100

Expected behavior
All DAGs present in the serialized_dag table should be ingested and displayed in the OpenMetadata UI, not just the top 100.

Version:

• OS: MacOS
• Python version:
• OpenMetadata version: 1.5.5
• OpenMetadata Ingestion package version: docker.getcollate.io/openmetadata/ingestion:1.5.5

Additional context
This issue persists even after updating the Airflow MySQL database to version 8. The problem was reproducible using only OpenMetadata images. Further investigation is needed to determine if there is a hidden configuration parameter or a bug in the ingestion framework that limits the number of imported DAGs.