diff --git a/documentation/_static/airflow_home.png b/documentation/_static/airflow_home.png new file mode 100644 index 00000000000..8fe2e4c4bab Binary files /dev/null and b/documentation/_static/airflow_home.png differ diff --git a/documentation/_static/airflow_login.png b/documentation/_static/airflow_login.png new file mode 100644 index 00000000000..f2c8961db23 Binary files /dev/null and b/documentation/_static/airflow_login.png differ diff --git a/documentation/_static/dag_index.png b/documentation/_static/dag_index.png new file mode 100644 index 00000000000..5624f08b4e1 Binary files /dev/null and b/documentation/_static/dag_index.png differ diff --git a/documentation/_static/dag_unpaused.png b/documentation/_static/dag_unpaused.png new file mode 100644 index 00000000000..1c9397aa07e Binary files /dev/null and b/documentation/_static/dag_unpaused.png differ diff --git a/documentation/catalog/guides/dag_testing.md b/documentation/catalog/guides/dag_testing.md new file mode 100644 index 00000000000..471aec2253c --- /dev/null +++ b/documentation/catalog/guides/dag_testing.md @@ -0,0 +1,54 @@ +# Running DAGs locally + +[Apache Airflow](https://airflow.apache.org/) is used to manage our workflows +(DAGs) in the catalog. For more information see +[our quickstart guide](/catalog/guides/quickstart.md#api-data). This document +describes how to run or test out these DAGs locally during development. + +Additionally, it is worth noting that not all DAGs can be run locally in +development right away as some of them require API keys from the provider. +However, some other DAGs like the ones for SMK or Finnish Museums can be run +locally without any additional keys. In order to run some DAGs locally, you +might need to get the API keys from the provider and add them to the +`catalog/.env` file. + +## Getting started + +1. Refer to and follow the instructions at [Quickstart](./quickstart.md) to + setup and make sure the catalog service is up and running. If you have + successfully completed the general setup then this can be started by running + `just catalog/up`. + +2. Navigate to http://localhost:9090 + +![Airflow Login Page](/_static/airflow_login.png) + +3. You should be met with an authentication page, use `airflow` as both the + username and password to log in. + +![Airflow Home Page](/_static/airflow_home.png) + +4. Search for or scroll down to any DAG of choice that does not require an API + key and click on it. We are using the `finnish_museums_workflow` for this + example. + +![Airflow DAG Index](/_static/dag_index.png) + +5. Click the toggle button labelled "DAG" at the top left, an alert box pops up + for you to confirm the action, click "OK" to continue. A run will get kicked + off. + + DAGs which are run on a schedule (like this one) will usually kick off + immediately, though the page may need to be refreshed in order to view the + run. If you wish to kick off a new DAG run, you should click on the "Trigger + DAG" button represented by a "play" icon at the top right of the page. + +![Airflow DAG Unpaused](/_static/dag_unpaused.png) + +6. To get a summary of the DAG, click on the "DAG Docs" accordion and you should + see an overview of the DAG displayed. + +```{note} +For more info about how Airflow works in general, check out their +[documentation on the UI](https://airflow.apache.org/docs/apache-airflow/stable/ui.html). +``` diff --git a/documentation/catalog/guides/index.md b/documentation/catalog/guides/index.md index 97e94c5d686..3f57af7f192 100644 --- a/documentation/catalog/guides/index.md +++ b/documentation/catalog/guides/index.md @@ -7,4 +7,5 @@ quickstart deploy adding_a_new_provider deployment +dag_testing ``` diff --git a/documentation/meta/documentation/adding_docs.md b/documentation/meta/documentation/adding_docs.md new file mode 100644 index 00000000000..2ecdcf3dea0 --- /dev/null +++ b/documentation/meta/documentation/adding_docs.md @@ -0,0 +1,37 @@ +# Adding a new documentation page + +This is a quick guide for adding a new documentation page and running the +documentation locally. + +## Prerequisites + +Refer to the [general setup guide](/general/general_setup.md) for setting up the +prerequisites. Refer to the 'Docs' column in the +[requirement matrix](/general/general_setup.md#requirement-matrix) to know what +you need to run this. + +## Starting up + +The following demonstrates the process of adding a new documentation page. + +1. Create a markdown file at a place within the `documentation/` hierarchy that + makes the most sense for your case. Usually the parent folder for the new + file would already contain an `index.md` file, make sure that your new file + is located next to it. If a new directory is required, you will also need to + create an `index.md` file for it with a table of contents for the folder. See + other `index.md` files within the `documentation/` directory for examples. + + For instance, adding a new documentation file `dag_testing.md` for describing + how to test/run the catalog DAGs would likely have you adding the file in + `documentation/catalog/guides/`. + +2. Make an entry with the name of the new document (excluding the `.md` file + extension) in the closest `index.md`. For our `dag_testing.md` instance, this + would mean adding `dag_testing` to `documentation/catalog/guides/index.md`. + +3. Write the document using [MyST flavored Markdown](https://mystmd.org/guide). + +4. Run `just lint` to properly format the document. + +5. Run `just documentation/live` locally (following the quick start guide) and + navigate to the new page to make sure it looks okay! diff --git a/documentation/meta/documentation/index.md b/documentation/meta/documentation/index.md index 52872083963..62f8d5cfa74 100644 --- a/documentation/meta/documentation/index.md +++ b/documentation/meta/documentation/index.md @@ -5,6 +5,7 @@ quickstart guidelines +adding_docs extensions review ```