Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add existing Dagster project example to dbt quickstart #21817

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions docs/content/integrations/dbt/quickstart.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,84 @@ dagster-dbt project scaffold --project-name my_dagster_project --dbt-project-dir

This command creates a new directory called `my_dagster_project/` inside the current directory. The new `my_dagster_project/` directory will contain a set of files that define a Dagster project to load the dbt project provided in `./my_dbt_project`.

</TabItem>
<TabItem name="Option 2: Use an existing Dagster project">
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<TabItem name="Option 2: Use an existing Dagster project">
<TabItem name="Option 2: Load your dbt project in an existing Dagster project">

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in ce9d1a1


### Option 2: Load your dbt project in an existing Dagster project

You can use an existing Dagster project to run your dbt project. To do so, you'll need to add some objects using the [dagster-dbt library](/\_apidocs/libraries/dagster-dbt) and update your `Definitions` object. This example assumes that your existing Dagster project includes both `assets.py` and `definitions.py` files, among other required files.

```shell
my_dagster_project
├── __init__.py
├── assets.py
├── definitions.py
├── pyproject.toml
├── setup.cfg
└── setup.py
```

1. Change directories to the Dagster project directory:

```shell
cd my_dagster_project/
```

2. Create a Python file named `project.py` and add the following code. The DbtProject object is a representation of your dbt project that assist with managing `manifest.json` preparation.

```python file=/integrations/dbt/quickstart.py startafter=start_dbt_project_example endbefore=end_dbt_project_example
from pathlib import Path

from dagster_dbt import DbtProject

RELATIVE_PATH_TO_MY_DBT_PROJECT = "./my_dbt_project"

my_project = DbtProject(
project_dir=Path(__file__)
.joinpath("..", RELATIVE_PATH_TO_MY_DBT_PROJECT)
.resolve(),
)
```

3. In your `assets.py` file, add the following code. Using the `dbt_assets` decorator allows Dagster to create a definition for how to compute a set of dbt resources, described by a `manifest.json`.

```python file=/integrations/dbt/quickstart.py startafter=start_dbt_assets_example endbefore=end_dbt_assets_example
from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, dbt_assets

from .project import my_project

@dbt_assets(manifest=my_project.manifest_path)
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
yield from dbt.cli(["build"], context=context).stream()
```

4. In your `definitions.py` file, update your Definitions object to include the newly created objects.

```python file=/integrations/dbt/quickstart.py startafter=start_dbt_definitions_example endbefore=end_dbt_definitions_example
from dagster import Definitions
from dagster_dbt import DbtCliResource

from .assets import my_dbt_assets
from .project import my_project

defs = Definitions(
...,
assets=[
...,
# Add the dbt assets alongside your other asset
my_dbt_assets
],
resources={
...,
# Add the dbt resource alongside your other resources
"dbt": DbtCliResource(project_dir=my_project),
},
)
```

With these changes, your existing Dagster project is ready to run your dbt project.

</TabItem>
</TabGroup>

Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# pyright: reportMissingImports=false
# ruff: isort: skip_file

# start_example
Expand Down Expand Up @@ -40,3 +41,60 @@ def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
},
)
# end_example


# start_dbt_project_example
from pathlib import Path

from dagster_dbt import DbtProject

RELATIVE_PATH_TO_MY_DBT_PROJECT = "./my_dbt_project"

my_project = DbtProject(
project_dir=Path(__file__)
.joinpath("..", RELATIVE_PATH_TO_MY_DBT_PROJECT)
.resolve(),
)
# end_dbt_project_example


# Using a string to avoid ruff adding a second blank line before the dbt_assets.
"""
# start_dbt_assets_example
from dagster import AssetExecutionContext
from dagster_dbt import DbtCliResource, dbt_assets

from .project import my_project

@dbt_assets(manifest=my_project.manifest_path)
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
yield from dbt.cli(["build"], context=context).stream()
# end_dbt_assets_example
"""


# Using a string to avoid compilation error caused by the `...` placeholders
"""# start_dbt_definitions_example

from dagster import Definitions
from dagster_dbt import DbtCliResource

from .assets import my_dbt_assets
from .project import my_project

defs = Definitions(
...,
assets=[
...,
# Add the dbt assets alongside your other asset
my_dbt_assets
],
resources={
...,
# Add the dbt resource alongside your other resources
"dbt": DbtCliResource(project_dir=my_project),
},
)

# end_dbt_definitions_example
"""
Loading