Move filtered index creation totally to Airflow #3240
Labels
💻 aspect: code
Concerns the software code in the repository
✨ goal: improvement
Improvement to an existing user-facing feature
🟨 priority: medium
Not blocking but should be addressed soon
🧱 stack: catalog
Related to the catalog and Airflow DAGs
🧱 stack: ingestion server
Related to the ingestion/data refresh server
⛔ status: blocked
Blocked & therefore, not ready for work
🔧 tech: airflow
Involves Apache Airflow
🐍 tech: python
Involves Python
Problem
Currently, the DAGs for the creation of filtered indexes (for image and audio) depend on the Ingestion Server. There is no reason we can not leave all that work to Airflow and it would be preferable to have fewer moving parts so it's also easier to debug when things go wrong.
Description
Move the
create_and_populate_filtered_index
function out of the Ingestion Server to the create filtered index DAG in the Catalog.openverse/ingestion_server/ingestion_server/indexer.py
Lines 465 to 471 in 41a1272
Additional context
This will be required down the line for other DAGs in the Search relevancy sandbox project.
The text was updated successfully, but these errors were encountered: