forked from dagster-io/dagster
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix(dbt): support projects with saved queries (dagster-io#21441)
## Summary & Motivation When i upgrade from dagster 1.6.11 to 1.7.3 I am no longer able to use [dbt's saved queries](https://docs.getdbt.com/docs/build/saved-queries) via the [dagster-dbt integration](https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-dbt). I am now getting a `key error` before any definitions can be loaded by dagster from the dagster-dbt assets example: <img width="1215" alt="image" src="https://github.com/dagster-io/dagster/assets/24627926/231c7e6a-e1aa-4a11-af83-d25853a7a97a"> For reference `saved_queries` are located in the [dbt's manifest.json schema](https://schemas.getdbt.com/dbt/manifest/v11/index.html#saved_queries) ## The problem Manifest creation in master (dagster 1.7.3): https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/utils.py#L255 has changed how the manifest is parsed and no longer picks up saved_queries. Manifest creation in master (dagster 1.6.11): https://github.com/dagster-io/dagster/blob/release-1.6.11/python_modules/libraries/dagster-dbt/dagster_dbt/utils.py#L243 is the version that currently works for me. It simply gets all keys from dbt's manifest.json with the code`Manifest.from_dict(manifest_json)` I have added the `saved_queries` key to this branch to fix this up ## How I Tested These Changes Have tested this locally (editable install) and it allows me to load all definitions including dbt's saved queries.
- Loading branch information
1 parent
b6704f8
commit 998678b
Showing
4 changed files
with
36 additions
and
17 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
23 changes: 23 additions & 0 deletions
23
...dagster_dbt_tests/dbt_projects/test_dagster_dbt_semantic_models/models/semantic_model.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
version: 2 | ||
|
||
semantic_models: | ||
- name: customers | ||
description: Customer grain mart. | ||
model: ref('customers') | ||
entities: | ||
- name: customer | ||
expr: customer_id | ||
type: primary | ||
measures: | ||
- name: total_order_amount | ||
description: Total count of orders per customer. | ||
agg: sum | ||
- name: total_order_amount | ||
agg: sum | ||
description: Gross customer lifetime spend inclusive of taxes. | ||
|
||
saved_queries: | ||
- name: customers_query | ||
description: Customer query | ||
label: Customer query | ||
query_params: {} |