Skip to content

Commit

Permalink
fix(dbt): support projects with saved queries (#21441)
Browse files Browse the repository at this point in the history
## Summary & Motivation

When i upgrade from dagster 1.6.11 to 1.7.3 I am no longer able to use [dbt's saved queries](https://docs.getdbt.com/docs/build/saved-queries) via the [dagster-dbt integration](https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-dbt). I am now getting a `key error` before any definitions can be loaded by dagster from the dagster-dbt assets example:

<img width="1215" alt="image" src="https://github.com/dagster-io/dagster/assets/24627926/231c7e6a-e1aa-4a11-af83-d25853a7a97a">


For reference `saved_queries` are located in the [dbt's manifest.json schema](https://schemas.getdbt.com/dbt/manifest/v11/index.html#saved_queries)

## The problem

Manifest creation in master (dagster 1.7.3): https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/utils.py#L255 has changed how the manifest is parsed and no longer picks up saved_queries.


Manifest creation in master (dagster 1.6.11): https://github.com/dagster-io/dagster/blob/release-1.6.11/python_modules/libraries/dagster-dbt/dagster_dbt/utils.py#L243 is the version that currently works for me. It simply gets all keys from dbt's manifest.json with the code`Manifest.from_dict(manifest_json)`


I have added the `saved_queries` key to this branch to fix this up

## How I Tested These Changes

Have tested this locally (editable install) and it allows me to load all definitions including dbt's saved queries.
  • Loading branch information
wisemuffin authored Apr 30, 2024
1 parent 9d97ffa commit 7d0cf6d
Show file tree
Hide file tree
Showing 4 changed files with 36 additions and 17 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -641,6 +641,7 @@ def is_non_asset_node(dbt_resource_props: Mapping[str, Any]):
[
resource_type == "metric",
resource_type == "semantic_model",
resource_type == "saved_query",
resource_type == "model"
and dbt_resource_props.get("config", {}).get("materialized") == "ephemeral",
]
Expand Down
13 changes: 12 additions & 1 deletion python_modules/libraries/dagster-dbt/dagster_dbt/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -273,12 +273,22 @@ def __getattr__(self, item):
{
"semantic_models": {
unique_id: _DictShim(info)
for unique_id, info in manifest_json.get("semantic_models", {}).items()
for unique_id, info in manifest_json["semantic_models"].items()
}
}
if manifest_json.get("semantic_models")
else {}
),
**(
{
"saved_queries": {
unique_id: _DictShim(info)
for unique_id, info in manifest_json["saved_queries"].items()
},
}
if manifest_json.get("saved_queries")
else {}
),
)
child_map = manifest_json["child_map"]

Expand Down Expand Up @@ -313,6 +323,7 @@ def get_dbt_resource_props_by_dbt_unique_id_from_manifest(
**manifest["exposures"],
**manifest["metrics"],
**manifest.get("semantic_models", {}),
**manifest.get("saved_queries", {}),
}


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,19 +81,3 @@ models:
description: Amount of the order (AUD) paid for by gift card
tests:
- not_null

semantic_models:
- name: customers
description: Customer grain mart.
model: ref('customers')
entities:
- name: customer
expr: customer_id
type: primary
measures:
- name: total_order_amount
description: Total count of orders per customer.
agg: sum
- name: total_order_amount
agg: sum
description: Gross customer lifetime spend inclusive of taxes.
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
version: 2

semantic_models:
- name: customers
description: Customer grain mart.
model: ref('customers')
entities:
- name: customer
expr: customer_id
type: primary
measures:
- name: total_order_amount
description: Total count of orders per customer.
agg: sum
- name: total_order_amount
agg: sum
description: Gross customer lifetime spend inclusive of taxes.

saved_queries:
- name: customers_query
description: Customer query
label: Customer query
query_params: {}

0 comments on commit 7d0cf6d

Please sign in to comment.