Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The alter data step of the new image data refresh process could not finish #5290

Open
krysal opened this issue Dec 18, 2024 · 0 comments
Open
Labels
💻 aspect: code Concerns the software code in the repository 🛠 goal: fix Bug fix 🟧 priority: high Stalls work on the project or its dependents python Pull requests that update Python code 🧱 stack: catalog Related to the catalog and Airflow DAGs 🔧 tech: airflow Involves Apache Airflow

Comments

@krysal
Copy link
Member

krysal commented Dec 18, 2024

Description

The alter_data_batch of the new staging_image_data_refresh DAG causes the Airflow instance to crash. #5145 attempted to fix it by increasing the size of batches, creating fewer tasks to expand, but the result didn't change. This process consumes a lot of the Airflow instance memory, even with the number of active tasks restricted to 2 (#5125), so it needs to be optimized. An alternative is to convert it to an iterative task, similar to how the batched_update operates.

We know the rest of the steps work, given that the staging_audio_data_refresh DAG ran successfully, but the alter process is exclusive to the image table.

Reproduction

Take special care and monitor closely when testing this DAG.

  1. Pause the legacy data refresh DAGs once the process finishes
  2. Unpause the staging_image_data_refresh and trigger it
  3. The UI will start to be inaccessible, so the DAG must be stopped when it is up again or via Airflow CLI.

Additional context

Part of #3925.

@krysal krysal added python Pull requests that update Python code 💻 aspect: code Concerns the software code in the repository 🔧 tech: airflow Involves Apache Airflow 🛠 goal: fix Bug fix 🟧 priority: high Stalls work on the project or its dependents 🧱 stack: catalog Related to the catalog and Airflow DAGs labels Dec 18, 2024
@openverse-bot openverse-bot moved this to 📋 Backlog in Openverse Backlog Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💻 aspect: code Concerns the software code in the repository 🛠 goal: fix Bug fix 🟧 priority: high Stalls work on the project or its dependents python Pull requests that update Python code 🧱 stack: catalog Related to the catalog and Airflow DAGs 🔧 tech: airflow Involves Apache Airflow
Projects
Status: 📋 Backlog
Development

No branches or pull requests

1 participant