Skip to content

Commit

Permalink
Add dbt quickstart example for already existing Dagster project
Browse files Browse the repository at this point in the history
  • Loading branch information
maximearmstrong committed May 13, 2024
1 parent 0ddd004 commit 58b6518
Show file tree
Hide file tree
Showing 2 changed files with 109 additions and 0 deletions.
61 changes: 61 additions & 0 deletions docs/content/integrations/dbt/quickstart.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,67 @@ dagster-dbt project scaffold --project-name my-dagster-project --dbt-project-dir

This creates a directory called `my-dagster-project/` inside the current directory. The `my-dagster-project/` directory contains a set of files that define a Dagster project.

</TabItem>
<TabItem name="Option 2: Use an existing Dagster project">

### Option 2: Use an existing Dagster project

You can use an existing Dagster project to run your dbt project. To do so, you'll need to add some objects using the [dagster-dbt library](/\_apidocs/libraries/dagster-dbt) and update your `Definitions` object. This example assumes that your existing Dagster project includes both `assets.py` and `definitions.py` files.

1. Change directories to the Dagster project directory:

```shell
cd my-dagster-project/
```

2. Create a Python file named `project.py` and add the following code. The DbtProject object is a representation of your dbt project that assist with managing manifest.json preparation.

```python file=/integrations/dbt/quickstart.py startafter=start_dbt_project_example endbefore=end_dbt_project_example
from pathlib import Path

from dagster_dbt import DbtProject

RELATIVE_PATH_TO_MY_DBT_PROJECT = "./my-dbt-project"

my_project = DbtProject(
project_dir=Path(__file__)
.joinpath("..", RELATIVE_PATH_TO_MY_DBT_PROJECT)
.resolve(),
)
```

3. In your `assets.py` file, add the following code. Using the `dbt_assets` decorator allows Dagster to create a definition for how to compute a set of dbt resources, described by a manifest.json.

```python file=/integrations/dbt/quickstart.py startafter=start_dbt_assets_example endbefore=end_dbt_assets_example
from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, dbt_assets

from .project import my_project

@dbt_assets(manifest=my_project.manifest_path)
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
yield from dbt.cli(["build"], context=context).stream()
```

4. In your `definitions.py` file, update your Definitions object to include the newly created objects.

```python file=/integrations/dbt/quickstart.py startafter=start_dbt_definitions_example endbefore=end_dbt_definitions_example
from dagster import Definitions
from dagster_dbt import DbtCliResource

from .assets import my_dbt_assets
from .project import my_project

defs = Definitions(
assets=[my_dbt_assets],
resources={
"dbt": DbtCliResource(project_dir=my_project),
},
)
```

With these changes, your existing Dagster project is ready to run your dbt project.

</TabItem>
</TabGroup>

Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# pyright: reportMissingImports=false
# ruff: isort: skip_file

# start_example
Expand Down Expand Up @@ -40,3 +41,50 @@ def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
},
)
# end_example


def dbt_project_example():
# start_dbt_project_example
from pathlib import Path

from dagster_dbt import DbtProject

RELATIVE_PATH_TO_MY_DBT_PROJECT = "./my-dbt-project"

my_project = DbtProject(
project_dir=Path(__file__)
.joinpath("..", RELATIVE_PATH_TO_MY_DBT_PROJECT)
.resolve(),
)
# end_dbt_project_example


def dbt_assets_example():
# start_dbt_assets_example
from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, dbt_assets

from .project import my_project

@dbt_assets(manifest=my_project.manifest_path)
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
yield from dbt.cli(["build"], context=context).stream()

# end_dbt_assets_example


def dbt_definitions_example():
# start_dbt_definitions_example
from dagster import Definitions
from dagster_dbt import DbtCliResource

from .assets import my_dbt_assets
from .project import my_project

defs = Definitions(
assets=[my_dbt_assets],
resources={
"dbt": DbtCliResource(project_dir=my_project),
},
)
# end_dbt_definitions_example

0 comments on commit 58b6518

Please sign in to comment.