Skip to content

Commit

Permalink
Update dbt source post review
Browse files Browse the repository at this point in the history
  • Loading branch information
maximearmstrong committed Aug 5, 2024
1 parent 742f00f commit 54fd52f
Show file tree
Hide file tree
Showing 5 changed files with 16 additions and 18 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,17 @@ lesson: '3'

As you’ll frequently point your Dagster code to the `target/manifest.json` file and your dbt project in this course, it’ll be helpful to keep a reusable representation of the dbt project. This can be easily done using the `DbtProject` class.

Create a new file in the `assets` directory called `dbt.py`. Open that file and add the following import at the top:
In the `assets` directory, create a new `dbt.py` file and add the following imports:

```python
from pathlib import Path

from dagster_dbt import DbtProject
```

The `Path` class from the `pathlib` standard library will help us create an accurate pointer to where our dbt project is. The `DbtProject` class is imported from the `dagster_dbt` package that we installed earlier. After the import, add the following code:
The `Path` class from the `pathlib` standard library will help us create an accurate pointer to where our dbt project is. The `DbtProject` class is imported from the `dagster_dbt` package that we installed earlier.

After the import, add the following code:

```python
dbt_project = DbtProject(
Expand All @@ -26,7 +28,7 @@ dbt_project = DbtProject(

This code creates a representation of the dbt project called `dbt_project`. The code defining the location of the project directory might look a little complicated, so let’s break it down:

- It uses the location of the `dbt.py` file (via `__file__`) as a point of reference for finding the dbt project
- The location of the `dbt.py` file (via `__file__`) is used as a point of reference for finding the dbt project
- The arguments in `joinpath` point us towards our dbt project by appending the following to the current path:
- Three directory levels up (`"..", "..", ".."`)
- A directory named `analytics`, which is the directory containing our dbt project
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ We’ll only create one `@dbt_assets` definition for now, but in a later lesson,

## Loading the models as assets

1. Navigate to the `assets/dbt.py` created earlier and open it.
1. Open the `assets/dbt.py` file.

2. Add the following imports to the top of the file:

Expand All @@ -37,13 +37,7 @@ We’ll only create one `@dbt_assets` definition for now, but in a later lesson,
from dagster_dbt import dbt_assets, DbtCliResource
```

3. The `@dbt_assets` decorator requires a path to the project’s manifest file, which is within our `dbt_project` representation created earlier. The manifest path can be easily accessed with the code below:

```python
dbt_project.manifest_path
```

4. Now, use the `@dbt_assets` decorator to create a new asset function and provide it with a reference to the manifest:
3. Next, we'll use the `@dbt_assets` decorator to create a new asset function and provide it with a reference to the project's manifest file:

```python
@dbt_assets(
Expand All @@ -52,7 +46,9 @@ We’ll only create one `@dbt_assets` definition for now, but in a later lesson,
def dbt_analytics(context: AssetExecutionContext, dbt: DbtCliResource):
```

5. Finally, add the following to the body of `dbt_analytics` function:
Here, we used `dbt_project.manifest_path` to provide the reference to the project's manifest file. This is possible because the `dbt_project` representation we created earlier contains the manifest path, accessible by using the `manifest_path` attribute.

4. Finally, add the following to the body of `dbt_analytics` function:

```python
yield from dbt.cli(["run"], context=context).stream()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ lesson: '4'

By now, you’ve had to run `dbt parse` to create the manifest file and reload your code location quite frequently, which doesn’t feel like the cleanest developer experience.

Before we move on, we’ll reduce the number of steps in the feedback loop. We'll automate the creation the manifest file by taking advantage of the `DbtProject` that we wrote earlier.
Before we move on, we’ll reduce the number of steps in the feedback loop. We'll automate the creation of the manifest file by taking advantage of the `dbt_project` representation that we wrote earlier.

---

Expand All @@ -22,14 +22,14 @@ In `dbt.py`, after the code initializing `dbt_project`, add the following code:
dbt_project.prepare_if_dev()
```

If you look at the dbt project’s `/target` directory, you’ll see it stores the artifacts. When reloading your code in local development, when using `dagster_dev`, you'll see that a new manifest file is generated.
If you look at the dbt project’s `/target` directory, you’ll see it stores the artifacts. When you use `dagster dev` in local development and you reload your code, you'll see that a new manifest file is generated.

Reload your code location in the Dagster UI, and you’ll see that everything should still work: the dbt models are still shown as assets and you can manually materialize any of the models. The key difference is that you no longer have to manually run `dbt parse` anymore!

---

## Creating the manifest for production

This is great, however, it only handles the preparation of a new manifest file in local development. In production, where a dbt project is stable, we may want to prepare a new manifest file only at build time, during the deployment process. This can be done using the command line interface (CLI) available in the `dagster_dbt` package installed earlier.
This is great, however, it only handles the preparation of a new manifest file in local development. In production, where a dbt project is stable, we may want to prepare a new manifest file only at build time, during the deployment process. This can be done using the command line interface (CLI) available in the `dagster_dbt` package.

Don't worry about the details for now! In Lesson 7, we’ll discuss the details on how to create a manifest file programmatically during deployment using the `dagster_dbt` CLI.
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Building your manifest for your production deployment will be needed for both op

Since your CI/CD will be running in a fresh environment, you'll need to install dbt and other dependencies before building your manifest.

To get our deployment working, we need to add a step to our GitHub Actions workflow that runs the commands required to generate the `manifest.json`. Specifically, we need to run the `dbt project prepare-and-package` command, available in the `dagster_dbt` package discussed earlier.
To get our deployment working, we need to add a step to our GitHub Actions workflow that runs the commands required to generate the `manifest.json`. Specifically, we need to run the `dbt project prepare-and-package` command, available in the `dagster_dbt` package.

1. In your Dagster project, locate the `.github/workflows` directory.
2. Open the `deploy.yml` file.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -90,9 +90,9 @@ dbt_project = DbtProject(

## Adding a prod target to deploy.yml

After that, we need to update the dbt commands in the `.github/workflows/deploy.yml` file to target the new `prod` profile. This will ensure that dbt uses the correct connection details when the GitHub Action runs as part of our Dagster+ deployment.
Next, we need to update the dbt commands in the `.github/workflows/deploy.yml` file to target the new `prod` profile. This will ensure that dbt uses the correct connection details when the GitHub Action runs as part of our Dagster+ deployment.

Open the file, scroll to the environment variable sections, and set an environment variable named `DBT_TARGET` to `prod`. This command should be on or around line 12:
Open the file, scroll to the environment variable section, and set an environment variable named `DBT_TARGET` to `prod`. This should be on or around line 12:

```bash
env:
Expand Down

0 comments on commit 54fd52f

Please sign in to comment.