Skip to content

Commit

Permalink
Update course after moving DbtProject to project.py
Browse files Browse the repository at this point in the history
  • Loading branch information
maximearmstrong committed Aug 5, 2024
1 parent 5db629a commit 305f9ff
Show file tree
Hide file tree
Showing 8 changed files with 17 additions and 29 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ lesson: '3'

As you’ll frequently point your Dagster code to the `target/manifest.json` file and your dbt project in this course, it’ll be helpful to keep a reusable representation of the dbt project. This can be easily done using the `DbtProject` class.

In the `assets` directory, create a new `dbt.py` file and add the following imports:
In the `dagster_university` directory, create a new `project.py` file and add the following imports:

```python
from pathlib import Path
Expand All @@ -22,7 +22,7 @@ After the import, add the following code:

```python
dbt_project = DbtProject(
project_dir=Path(__file__).joinpath("..", "..", "..", "analytics").resolve(),
project_dir=Path(__file__).joinpath("..", "..", "analytics").resolve(),
)
```

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Navigate to the `dagster_university/resources/__init__.py`, which is where other
```python
from dagster_dbt import DbtCliResource

from ..assets.dbt import dbt_project
from ..project import dbt_project
# the import lines go at the top of the file

# this can be defined anywhere below the imports
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,13 +28,15 @@ We’ll only create one `@dbt_assets` definition for now, but in a later lesson,

## Loading the models as assets

1. Open the `assets/dbt.py` file.
1. Create a new file in the `assets` directory called `dbt.py`.

2. Add the following imports to the top of the file:

```python
from dagster import AssetExecutionContext
from dagster_dbt import dbt_assets, DbtCliResource

from ..project import dbt_project
```

3. Next, we'll use the `@dbt_assets` decorator to create a new asset function and provide it with a reference to the project's manifest file:
Expand Down Expand Up @@ -66,14 +68,10 @@ We’ll only create one `@dbt_assets` definition for now, but in a later lesson,
At this point, `dbt.py` should look like this:

```python
from pathlib import Path

from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, DbtProject, dbt_assets
from dagster_dbt import DbtCliResource, dbt_assets

dbt_project = DbtProject(
project_dir=Path(__file__).joinpath("..", "..", "..", "analytics").resolve(),
)
from ..project import dbt_project


@dbt_assets(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Before we move on, we’ll reduce the number of steps in the feedback loop. We'l

The first detail is that the `dbt_project` doesn’t need to be part of an asset to be executed. This means that once a `dbt_project` is defined, you can use it to execute commands when your code location is being built. Rather than manually running `dbt parse`, let’s use the `dbt_project` to prepare the manifest file for us.

In `dbt.py`, after the code initializing `dbt_project`, add the following code:
In `project.py`, after the code initializing `dbt_project`, add the following code:

```python
dbt_project.prepare_if_dev()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -105,15 +105,10 @@ Open the `assets/dbt.py` file and do the following:
At this point, your `dbt.py` file should match the following:

```python
from pathlib import Path

from dagster import AssetExecutionContext, AssetKey
from dagster_dbt import DagsterDbtTranslator, DbtCliResource, DbtProject, dbt_assets
from dagster_dbt import DagsterDbtTranslator, DbtCliResource, dbt_assets

dbt_project = DbtProject(
project_dir=Path(__file__).joinpath("..", "..", "..", "analytics").resolve(),
)
dbt_project.prepare_if_dev()
from ..project import dbt_project


class CustomizedDagsterDbtTranslator(DagsterDbtTranslator):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -122,20 +122,15 @@ At this point, the `dagster_university/assets/dbt.py` file should look like this

```python
import json
from pathlib import Path

from dagster import AssetExecutionContext, AssetKey
from dagster_dbt import DagsterDbtTranslator, DbtCliResource, DbtProject, dbt_assets
from dagster_dbt import DagsterDbtTranslator, DbtCliResource, dbt_assets

from ..partitions import daily_partition
from ..project import dbt_project

INCREMENTAL_SELECTOR = "config.materialized:incremental"

dbt_project = DbtProject(
project_dir=Path(__file__).joinpath("..", "..", "..", "analytics").resolve(),
)
dbt_project.prepare_if_dev()


class CustomizedDagsterDbtTranslator(DagsterDbtTranslator):
def get_asset_key(self, dbt_resource_props):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ To get our deployment working, we need to add a step to our GitHub Actions workf
pip install pip --upgrade
cd project-repo
pip install . --upgrade --upgrade-strategy eager
dagster-dbt project prepare-and-package --file dagster_university/assets/dbt.py
dagster-dbt project prepare-and-package --file dagster_university/project.py
shell: bash
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,15 +63,15 @@ Because we’re still using a DuckDB-backed database, our `type` will also be `d

## Adding a prod target to DbtProject

Next, we need to update the `DbtProject` object in `dagster_university/assets/dbt.py` to specify what profile to target. To optimize the developer experience, let’s use an environment variable to specify the profile to target.
Next, we need to update the `DbtProject` object in `dagster_university/project.py` to specify what profile to target. To optimize the developer experience, let’s use an environment variable to specify the profile to target.

1. In the `.env` file, define an environment variable named `DBT_TARGET` and set it to `dev`:

```python
DBT_TARGET=dev
```

2. Next, import the `os` module at the top of the `dbt.py` file so the environment variable is accessible:
2. Next, import the `os` module at the top of the `project.py` file so the environment variable is accessible:

```python
import os
Expand All @@ -81,7 +81,7 @@ Next, we need to update the `DbtProject` object in `dagster_university/assets/db

```python
dbt_project = DbtProject(
project_dir=Path(__file__).joinpath("..", "..", "..", "analytics").resolve(),
project_dir=Path(__file__).joinpath("..", "..", "analytics").resolve(),
target=os.getenv("DBT_TARGET")
)
```
Expand Down

0 comments on commit 305f9ff

Please sign in to comment.