Skip to content

Commit

Permalink
[docs][dagster-dbt] use new with_insights chain method (#23203)
Browse files Browse the repository at this point in the history
## Summary

Adds docs for the new insights API introduced in #23183
  • Loading branch information
benpankow authored Aug 6, 2024
1 parent 5d7099e commit b63ebae
Show file tree
Hide file tree
Showing 6 changed files with 15 additions and 35 deletions.
Binary file modified docs/content/api/modules.json.gz
Binary file not shown.
Binary file modified docs/content/api/searchindex.json.gz
Binary file not shown.
Binary file modified docs/content/api/sections.json.gz
Binary file not shown.
Original file line number Diff line number Diff line change
Expand Up @@ -35,39 +35,29 @@ To complete the steps in this guide, you'll need:
<TabGroup>
<TabItem name="Using assets">

First, instrument the Dagster <PyObject module="dagster_dbt" object="dbt_assets" decorator /> function with `dbt_with_bigquery_insights`:
First, append <PyObject object="with_insights" module="dagster_dbt.core.dbt_cli_invocation.DbtEventIterator" displayText="with_insights()" /> to the dbt CLI call in your Dagster <PyObject module="dagster_dbt" object="dbt_assets" decorator /> function:

```python
from dagster_cloud.dagster_insights import dbt_with_bigquery_insights


@dbt_assets(...)
def my_asset(context: AssetExecutionContext, dbt: DbtCliResource):
# Typically you have a `yield from dbt_resource.cli(...)`.
# Wrap the original call with `dbt_with_bigquery_insights` as below.
dbt_cli_invocation = dbt_resource.cli(["build"], context=context)
yield from dbt_with_bigquery_insights(context, dbt_cli_invocation)
# Chain `with_insights` after any other metadata fetch, e.g. `fetch_row_count`
yield from dbt_resource.cli(["build"], context=context).stream().with_insights()
```

This passes through all underlying events and emits additional <PyObject object="AssetObservation" pluralize /> with BigQuery cost metrics. These metrics are obtained by querying the underlying `INFORMATION_SCHEMA.JOBS` table, using the BigQuery client from the dbt adapter.

</TabItem>
<TabItem name="Using ops and jobs">

First, instrument the op function with `dbt_with_bigquery_insights`:
First, append <PyObject object="with_insights" module="dagster_dbt.core.dbt_cli_invocation.DbtEventIterator" displayText="with_insights()" /> to the dbt CLI call in your Dagster op function:

```python
from dagster_cloud.dagster_insights import dbt_with_bigquery_insights


@op(out={})
def my_dbt_op(context: OpExecutionContext, dbt: DbtCliResource):
# Typically you have a `yield from dbt_resource.cli(...)`.
# Wrap the original call with `dbt_with_bigquery_insights` as below.
dbt_cli_invocation = dbt.cli(
# Chain `with_insights` after any other metadata fetch, e.g. `fetch_row_count`
yield from dbt.cli(
["build"], context=context, manifest=dbt_manifest_path
)
yield from dbt_with_bigquery_insights(context, dbt_cli_invocation)
).stream().with_insights()

@job
def my_dbt_job():
Expand Down Expand Up @@ -110,7 +100,7 @@ width={3454}
height={1338}
/>

The BigQuery cost metric is based off of the bytes billed for queries wrapped with `dbt_with_bigquery_insights`, based on a unit price of $6.25 USD per TiB.
The BigQuery cost metric is based off of the bytes billed for queries wrapped with `with_insights`, based on a unit price of $6.25 USD per TiB.

---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,39 +35,29 @@ To complete the steps in this guide, you'll need:
<TabGroup>
<TabItem name="Using assets">

First, instrument the Dagster <PyObject module="dagster_dbt" object="dbt_assets" decorator /> function with `dbt_with_snowflake_insights`:
First, append <PyObject object="with_insights" module="dagster_dbt.core.dbt_cli_invocation.DbtEventIterator" displayText="with_insights()" /> to the dbt CLI call in your Dagster <PyObject module="dagster_dbt" object="dbt_assets" decorator /> function:

```python
from dagster_cloud.dagster_insights import dbt_with_snowflake_insights


@dbt_assets(...)
def my_asset(context: AssetExecutionContext, dbt: DbtCliResource):
# Typically you have a `yield from dbt_resource.cli(...)`.
# Wrap the original call with `dbt_with_snowflake_insights` as below.
dbt_cli_invocation = dbt_resource.cli(["build"], context=context)
yield from dbt_with_snowflake_insights(context, dbt_cli_invocation)
# Chain `with_insights` after any other metadata fetch, e.g. `fetch_row_count`
yield from dbt_resource.cli(["build"], context=context).stream().with_insights()
```

This passes through all underlying events and emits an <PyObject object="AssetObservation" /> for each asset materialization. The observation contains the dbt invocation ID and unique ID recorded in the Dagster event log.

</TabItem>
<TabItem name="Using ops and jobs">

First, instrument the op function with `dbt_with_snowflake_insights`:
First, append <PyObject object="with_insights" module="dagster_dbt.core.dbt_cli_invocation.DbtEventIterator" displayText="with_insights()" /> to the dbt CLI call in your Dagster op function:

```python
from dagster_cloud.dagster_insights import dbt_with_snowflake_insights


@op(out={})
def my_dbt_op(context: OpExecutionContext, dbt: DbtCliResource):
# Typically you have a `yield from dbt_resource.cli(...)`.
# Wrap the original call with `dbt_with_snowflake_insights` as below.
dbt_cli_invocation = dbt.cli(
# Chain `with_insights` after any other metadata fetch, e.g. `fetch_row_count`
yield from dbt.cli(
["build"], context=context, manifest=dbt_manifest_path
)
yield from dbt_with_snowflake_insights(context, dbt_cli_invocation)
).stream().with_insights()

@job
def my_dbt_job():
Expand Down
Binary file modified docs/next/public/objects.inv
Binary file not shown.

1 comment on commit b63ebae

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs ready!

✅ Preview
https://dagster-docs-575803bc6-elementl.vercel.app
https://master.dagster.dagster-docs.io

Built with commit b63ebae.
This pull request is being automatically deployed with vercel-action

Please sign in to comment.