feat(dbt): emit column dependencies using `sqlglot` #20407

rexledesma · 2024-03-11T16:15:18Z

Summary & Motivation

Makes use of the great sqlglot library to build column lineage metadata when executing a dbt project.

We do this in the following steps:

Retrieve the current dbt node's SQL file and its parents' column schemas.
Retrieve the column names from the current node.
For each column, retrieve its dependencies on upstream columns from direct parents. Basically just invoke lineage from sqlglot)
Render the lineage as a JSON blob on the asset materialization for the dbt node.

To retrieve the dbt node's parents, and those corresponding nodes' column schemas, we augment our dagster dbt package implementation from #19623 to emit column schemas for the dbt node's parents. We make use of the dbt model variable to retrieve dbt node's refs/sources as relation objects to pass to adapter.get_columns_in_relation.

How I Tested These Changes

pytest

assert expected column dependencies against jaffle shop
assert expected column dependencies against executing a subset of jaffle shop
assert expected column dependencies against executing a subset of jaffle shop with ambiguous column selection (e.g. select *)

rexledesma · 2024-03-11T16:15:32Z

feat(dbt): add parameter toggle to emit column dependency metadata #20548 : 2 dependent PRs (#20477 , #20597 )
feat(dbt): emit column dependencies using sqlglot #20407 👈
test(dbt): remove tests against manually constructed dbt events #20531
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @rexledesma and the rest of your teammates on Graphite

sryza

If I understand correctly, this won't get lineage in the cases where we don't know the columns on the root nodes - is that right? Assuming the idea is to do that in a followup?

python_modules/libraries/dagster-dbt/dagster_dbt_tests/core/test_resources_v2.py

python_modules/libraries/dagster-dbt/dagster_dbt/core/resources_v2.py

python_modules/libraries/dagster-dbt/dbt_packages/dagster/macros/log_columns_in_relation.sql

python_modules/libraries/dagster-dbt/dagster_dbt/core/resources_v2.py

github-actions · 2024-03-18T20:39:47Z

Deploy preview for dagster-docs ready!

Preview available at https://dagster-docs-901u68c52-elementl.vercel.app
https://rl-column-lineage-metadata.dagster.dagster-docs.io

Direct link to changed pages:

sryza

Everything here looks great except for the MetadataValue stuff. I think life will be easier if we work that out before merging this. Otherwise, we'll need to deal with situations (annoying even if just internally) where we load data that's serialized in the old format.

Last thing is that I'm a little concerned about the perf impact. I don't think this is a blocker for merging, but I think it's worth seeing what the impact is in purina.

python_modules/libraries/dagster-dbt/dbt_packages/dagster/macros/log_column_level_metadata.sql

rexledesma · 2024-03-18T22:00:24Z

Everything here looks great except for the MetadataValue stuff. I think life will be easier if we work that out before merging this. Otherwise, we'll need to deal with situations (annoying even if just internally) where we load data that's serialized in the old format.

We always take the most recent materialization, so if serialization is a problem, we could just re-materialize to blow it away.

Last thing is that I'm a little concerned about the perf impact. I don't think this is a blocker for merging, but I think it's worth seeing what the impact is in purina.

We will dogfood this with our setup to see if there are latency concerns.

sryza

Nice. I think this should either get squashed with #20569 or they should be merged in close succession.

rexledesma · 2024-03-20T23:11:05Z

Merge activity

Mar 20, 4:11 PM PDT: @rexledesma started a stack merge that includes this pull request via Graphite.
Mar 20, 4:11 PM PDT: @rexledesma merged this pull request with Graphite.

## Summary & Motivation Apply #20477 on #20407 to see how the API feels. ## How I Tested These Changes pytest

## Summary & Motivation Makes use of the great `sqlglot` library to build column lineage metadata when executing a dbt project. We do this in the following steps: 1. Retrieve the current dbt node's SQL file and its parents' column schemas. 2. Retrieve the column names from the current node. 3. For each column, retrieve its dependencies on upstream columns from direct parents. Basically just invoke [`lineage`](https://sqlglot.com/sqlglot/lineage.html#lineage) from `sqlglot`) 4. Render the lineage as a JSON blob on the asset materialization for the dbt node. To retrieve the dbt node's parents, and those corresponding nodes' column schemas, we augment our `dagster` dbt package implementation from #19623 to emit column schemas for the dbt node's parents. We make use of the dbt [`model`](https://docs.getdbt.com/reference/dbt-jinja-functions/model) variable to retrieve dbt node's refs/sources as relation objects to pass to [`adapter.get_columns_in_relation`](https://docs.getdbt.com/reference/dbt-jinja-functions/adapter#get_columns_in_relation). ## How I Tested These Changes pytest - assert expected column dependencies against jaffle shop - assert expected column dependencies against executing a subset of jaffle shop - assert expected column dependencies against executing a subset of jaffle shop with ambiguous column selection (e.g. `select *`)

## Summary & Motivation Apply #20477 on #20407 to see how the API feels. ## How I Tested These Changes pytest

rexledesma force-pushed the rl/column-lineage-metadata branch 6 times, most recently from ac721fb to 69142c1 Compare March 12, 2024 15:01

rexledesma marked this pull request as ready for review March 14, 2024 16:33

rexledesma requested a review from sryza March 14, 2024 16:33

sryza reviewed Mar 14, 2024

View reviewed changes

rexledesma force-pushed the rl/column-lineage-metadata branch 2 times, most recently from c7308a2 to 6e08330 Compare March 17, 2024 18:29

rexledesma requested review from sryza and bengotow March 17, 2024 18:32

rexledesma force-pushed the rl/column-lineage-metadata branch 2 times, most recently from 5ba9174 to 55a2f82 Compare March 18, 2024 01:47

rexledesma changed the base branch from master to rl/remove-tests March 18, 2024 11:51

rexledesma mentioned this pull request Mar 18, 2024

test(dbt): remove tests against manually constructed dbt events #20531

Merged

rexledesma force-pushed the rl/column-lineage-metadata branch 4 times, most recently from ab841d7 to 9e61034 Compare March 18, 2024 17:21

sryza reviewed Mar 18, 2024

View reviewed changes

sryza mentioned this pull request Mar 18, 2024

[RFC] feat(metadata): add table spec metadata #20477

Closed

rexledesma requested a review from sryza March 18, 2024 20:35

rexledesma force-pushed the rl/column-lineage-metadata branch from 9e61034 to 9c2119a Compare March 18, 2024 20:35

rexledesma requested a review from erinkcochran87 as a code owner March 18, 2024 20:35

rexledesma changed the title ~~feat(dbt): emit column lineage using sqlglot~~ feat(dbt): emit column dependencies using sqlglot Mar 18, 2024

rexledesma force-pushed the rl/column-lineage-metadata branch from 9c2119a to c33f5da Compare March 18, 2024 21:06

sryza reviewed Mar 18, 2024

View reviewed changes

python_modules/libraries/dagster-dbt/dbt_packages/dagster/macros/log_column_level_metadata.sql Show resolved Hide resolved

rexledesma mentioned this pull request Mar 18, 2024

feat(dbt): add parameter toggle to emit column dependency metadata #20548

Merged

rexledesma requested a review from sryza March 18, 2024 22:05

Base automatically changed from rl/remove-tests to master March 18, 2024 22:06

rexledesma force-pushed the rl/column-lineage-metadata branch 2 times, most recently from 579cb49 to a953dac Compare March 19, 2024 19:43

rexledesma mentioned this pull request Mar 19, 2024

feat(dbt): use TableMetadataEntries to emit column lineage #20569

Merged

rexledesma force-pushed the rl/column-lineage-metadata branch from a953dac to 11709ae Compare March 19, 2024 20:45

rexledesma removed the request for review from erinkcochran87 March 19, 2024 21:40

rexledesma force-pushed the rl/column-lineage-metadata branch from 11709ae to 4f0105d Compare March 20, 2024 15:54

rexledesma mentioned this pull request Mar 20, 2024

[RFC] feat(metadata): add table spec metadata #20597

Merged

rexledesma force-pushed the rl/column-lineage-metadata branch 2 times, most recently from 0456137 to ba083cc Compare March 20, 2024 22:13

feat(dbt): emit column lineage using sqlglot

0628a9c

rexledesma force-pushed the rl/column-lineage-metadata branch from ba083cc to 0628a9c Compare March 20, 2024 22:39

sryza approved these changes Mar 20, 2024

View reviewed changes

rexledesma merged commit 5d0f8fd into master Mar 20, 2024
2 checks passed

rexledesma deleted the rl/column-lineage-metadata branch March 20, 2024 23:11

rexledesma added a commit that referenced this pull request Mar 20, 2024

feat(dbt): use TableMetadataEntries to emit column lineage (#20569)

6a62524

## Summary & Motivation Apply #20477 on #20407 to see how the API feels. ## How I Tested These Changes pytest

PedramNavid pushed a commit that referenced this pull request Mar 28, 2024

feat(dbt): use TableMetadataEntries to emit column lineage (#20569)

5e82663

## Summary & Motivation Apply #20477 on #20407 to see how the API feels. ## How I Tested These Changes pytest

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(dbt): emit column dependencies using `sqlglot` #20407

feat(dbt): emit column dependencies using `sqlglot` #20407

rexledesma commented Mar 11, 2024 •

edited

Loading

rexledesma commented Mar 11, 2024 •

edited

Loading

sryza left a comment

github-actions bot commented Mar 18, 2024 •

edited

Loading

sryza left a comment

rexledesma commented Mar 18, 2024

sryza left a comment

rexledesma commented Mar 20, 2024 •

edited

Loading

feat(dbt): emit column dependencies using sqlglot #20407

feat(dbt): emit column dependencies using sqlglot #20407

Conversation

rexledesma commented Mar 11, 2024 • edited Loading

Summary & Motivation

How I Tested These Changes

rexledesma commented Mar 11, 2024 • edited Loading

sryza left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 18, 2024 • edited Loading

sryza left a comment

Choose a reason for hiding this comment

rexledesma commented Mar 18, 2024

sryza left a comment

Choose a reason for hiding this comment

rexledesma commented Mar 20, 2024 • edited Loading

Merge activity

feat(dbt): emit column dependencies using `sqlglot` #20407

feat(dbt): emit column dependencies using `sqlglot` #20407

rexledesma commented Mar 11, 2024 •

edited

Loading

rexledesma commented Mar 11, 2024 •

edited

Loading

github-actions bot commented Mar 18, 2024 •

edited

Loading

rexledesma commented Mar 20, 2024 •

edited

Loading