Add dbt + Dagster+ guide (#21229)

## Summary & Motivation This PR adds a guide to help users getting started with dbt & Dagster+. Covered: - How to use "Import a dbt project" - includes an example with GitHub - How to use "Import a dbt & Dagster project" - includes another example with GitHub Cloud actions will be discussed more in PR #21950. ## How I Tested These Changes BK + local ``` make mdx-full-format make next-watch-build ``` --------- Co-authored-by: Erin Cochran <[email protected]>
dagster-io · Jun 7, 2024 · faaa6cf · faaa6cf · github-actions · Jun 7, 2024
1 parent dee425f
commit faaa6cf
Show file tree

Hide file tree

Showing 2 changed files with 224 additions and 0 deletions.
diff --git a/docs/content/_navigation.json b/docs/content/_navigation.json
@@ -885,6 +885,10 @@
               }
             ]
           },
+          {
+            "title": "Using dbt with Dagster+",
+            "path": "/integrations/dbt/using-dbt-with-dagster-plus"
+          },
           {
             "title": "Reference",
             "path": "/integrations/dbt/reference"

diff --git a/docs/content/integrations/dbt/using-dbt-with-dagster-plus.mdx b/docs/content/integrations/dbt/using-dbt-with-dagster-plus.mdx
@@ -0,0 +1,220 @@
+---
+title: "Using dbt with Dagster+ | Dagster Docs"
+description: Deploy your dbt & Dagster project in Dagster+.
+---
+
+# Using dbt with Dagster+
+
+Importing an existing dbt project in Dagster+ allows you to automatically load your dbt models as Dagster assets. This can be be done with:
+
+- An existing dbt project that is not already using Dagster, or
+- A Dagster project in which your dbt project is included
+
+In this guide, we'll demonstrate by using an existing dbt project that doesn't use Dagster.
+
+---
+
+## Prerequisites
+
+To follow the steps in this guide, you'll need **Dagster+ Organization Admin**, **Admin**, or **Editor** permissions. This is required to create a code location.
+
+You'll also need **an existing dbt project** that contains the following files in the repository root:
+
+- [`dbt_project.yml`](https://docs.getdbt.com/reference/dbt_project.yml)
+- [`profiles.yml`](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml)
+
+---
+
+## Step 1: Import your project in Dagster+
+
+<Note>
+  <strong>Heads up!</strong> This guide focuses on Serverless deployments in
+  Dagster+, but dbt projects may also be deployed if using a Hybrid deployment.
+  For more information about Hybrid deployments, refer to the{" "}
+  <a href="/dagster-plus/deployment/hybrid#hybrid-deployments-in-dagster">
+    Hybrid deployment documentation
+  </a>
+  .
+</Note>
+
+In this section, we'll demonstrate how to import an existing project to Dagster+. Our example imports the project from a GitHub repository, but Dagster+ also supports Gitlab.
+
+1. Sign in to your Dagster+ account.
+
+2. Navigate to **Deployment > Code locations**.
+
+3. Click **Add code location**.
+
+4. Depending on the type of project you imported, this step will vary:
+
+   - **For dbt-only projects**, click **Import a dbt project**, then **Continue**.
+   - **For dbt and Dagster projects,** click **Import a Dagster project**.
+
+5. At this point, you'll be prompted to select either GitHub or Gitlab. For this guide, we'll select **GitHub**.
+
+6. If prompted, sign into your GitHub account and complete the authorization process for the Dagster+ application. **Note**: The profile or organization you're using to authorize Dagster+ must have read and write access to the repository containing the project. After the authorization is complete, you'll be redirected back to Dagster+.
+
+7. In Dagster+, locate and select the repository containing the project by using the dropdowns. **Note**: dbt projects must have `dbt_profiles.yml` and `profiles.yml` files in the repository root or an error will display.
+
+8. Click **Continue** to begin the import process.
+
+9. The last step of the import process adds a few files, which we'll discuss in the next section, to the project. Depending on the type of project you imported, this step will vary:
+
+   - **For dbt-only projects**, Dagster+ will open a pull request to update the repository. You'll need to review and merge the pull request to complete the process.
+   - **For dbt and Dagster projects,** Dagster+ will directly commit the files to the repository.
+
+Once Dagster+ finishes importing the project, move onto the next step.
+
+---
+
+## Step 2: Review the repository changes
+
+The file structure of the repository will change the first time a project is deployed using Dagster+. For dbt projects, a few things will happen:
+
+- **A [`dagster_cloud.yaml` file](/dagster-plus/managing-deployments/dagster-cloud-yaml) will be created.** This file defines the project as a Dagster+ code location.
+- **GitHub Action files will be created in `.github/workflows`.** These files, named `branch_deployments.yml` and `deploy.yml`, manage the deployments of the repository.
+- **For dbt-only projects being deployed for the first time**, Dagster+ will create a new Dagster project in the repository using the [`dagster-dbt scaffold`](/integrations/dbt/reference#scaffolding-a-dagster-project-from-a-dbt-project) command. This will result in a Dagster project that matches the dbt project. For example, a dbt project named `my_dbt_project` will contain a Dagster project in `my_dbt_project/my_dbt_project` after the process completes. Refer to the [Dagster project files reference](/guides/understanding-dagster-project-files) to learn more about the files in a Dagster project.
+
+**Use the following tabs** to see how the repository will change for a [dbt-only project](#dbt-only-projects) and a [dbt and Dagster project](#dbt-and-dagster-projects) being deployed for the first time.
+
+<TabGroup>
+<TabItem name="dbt-only projects">
+
+### dbt-only projects
+
+<Note>
+  <strong>Looking for dbt and Dagster projects?</strong>{" "}
+  <a href="#dbt-and-dagster-projects">Click here!</a>
+</Note>
+
+Before the Dagster+ changes, a typical dbt project would include files like `dbt_project.yml`, `profiles.yml`, dbt models in `.sql` format, and sbt seeds in `.csv` format. As this is a git repository, other files like `.gitignore`, `LICENSE` and `README.md` may also be included:
+
+```shell
+## dbt-only project
+## before Dagster+ deployment
+
+my_dbt_project
+├── models
+│   ├── my_model.sql
+├── seeds
+│   ├── my_seeds.csv
+├── .gitignore
+├── LICENSE
+├── README.md
+├── dbt_project.yml
+└── profiles.yml
+```
+
+When the Dagster+ deployment process completes, the repository will now look like the following:
+
+```shell
+## dbt-only project
+## after Dagster+ deployment
+
+my_dbt_project
+├── .github                                                ## GitHub Action files
+│   ├── workflows
+│   │   ├── branch_deployments.yml
+│   │   ├── deploy.yml
+├── models
+│   ├── my_model.sql
+├── my_dbt_project                                         ## New Dagster project
+│   ├── my_dbt_project
+│   │   ├── __init__.py
+│   │   ├── assets.py
+│   │   ├── definitions.py
+│   │   ├── project.py
+│   │   ├── schedules.py
+│   ├── pyproject.toml
+│   ├── setup.py
+├── seeds
+│   ├── my_seeds.csv
+├── .gitignore
+├── LICENSE
+├── README.md
+├── dagster_cloud.yaml                                     ## Dagster+ code location file
+├── dbt_project.yml
+└── profiles.yml
+```
+
+</TabItem>
+<TabItem name="dbt and Dagster projects">
+
+### dbt and Dagster projects
+
+<Note>
+  <strong>Looking for dbt-only projects?</strong>{" "}
+  <a href="#dbt-only-projects">Click here!</a>
+</Note>
+
+After the Dagster+ changes, a dbt and Dagster project will include the files required for dbt and Dagster, some files related to git, and the newly-added Dagster+ files:
+
+```shell
+## dbt and Dagster project
+## after Dagster+ deployment
+
+my_dbt_and_dagster_project
+├── .github                                                ## GitHub Action files
+│   ├── workflows
+│   │   ├── branch_deployments.yml
+│   │   ├── deploy.yml
+├── dbt
+│   ├── models
+│   │   ├── my_model.sql
+│   ├── seeds
+│   │   ├── my_seeds.csv
+│   ├── dbt_project.yml
+│   ├── profiles.yml
+├── my_dbt_and_dagster_project
+│   ├── __init__.py
+│   ├── assets.py
+│   ├── definitions.py
+│   ├── project.py
+│   ├── schedules.py
+├── .gitignore
+├── LICENSE
+├── README.md
+├── dagster_cloud.yaml                                     ## Dagster+ code location file
+├── pyproject.toml
+└── setup.py
+```
+
+</TabItem>
+</TabGroup>
+
+---
+
+## What's next?
+
+For an end-to-end example, from the project creation to the deployment to Dagster+, check out the Dagster & dbt course in [Dagster University](https://courses.dagster.io).
+
+---
+
+## Related
+
+<ArticleList>
+  <ArticleListItem
+    title="dbt & Dagster"
+    href="/integrations/dbt"
+  ></ArticleListItem>
+  <ArticleListItem
+    title="dagster-dbt reference"
+    href="/integrations/dbt/reference"
+  ></ArticleListItem>
+  <ArticleListItem
+    title="Getting started with Dagster+"
+    href="/dagster-plus"
+  ></ArticleListItem>
+  <ArticleListItem
+    title="Dagster+ user roles and permissions"
+    href="/dagster-plus/account/managing-users/managing-user-roles-permissions"
+  ></ArticleListItem>
+  <ArticleListItem
+    title="dagster_cloud.yaml reference"
+    href="/dagster-plus/managing-deployments/dagster-cloud-yaml"
+  ></ArticleListItem>
+  <ArticleListItem
+    title="Dagster project files"
+    href="/guides/understanding-dagster-project-files"
+  ></ArticleListItem>
+</ArticleList>