feat(dbt): add parameter toggle to emit column dependency metadata #20548
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @rexledesma and the rest of your teammates on |
This was referenced Mar 18, 2024
rexledesma
changed the title
|
Deploy preview for dagster-docs ready! Preview available at https://dagster-docs-89ll1ol7y-elementl.vercel.app Direct link to changed pages: |
rexledesma
force-pushed
the
rl/column-lineage-metadata
branch
from
c33f5da to
579cb49
Compare
rexledesma
force-pushed
the
rl/make-column-dependencies-optional
branch
from
8c51d54 to
d035115
Compare
sryza
approved these changes
Mar 18, 2024
| @@ -570,6 +570,22 @@ snapshots: | |||
| - "{{ dagster.log_column_level_metadata() }}" | |||
| ``` | |||
|
|
|||
| Column dependencies can be removed from materialization metadata by disabling the collection of parent relation metadata. This can be done by setting the `enable_parent_relation_metadata_collection` argument to `False` in the `dagster.log_column_level_metadata()` macro: | |||
There was a problem hiding this comment.
Feel free to ignore if you feel otherwise, but collect_parent_relation_metadata is shorter and might be equally clear.
rexledesma
force-pushed
the
rl/column-lineage-metadata
branch
from
579cb49 to
a953dac
Compare
rexledesma
force-pushed
the
rl/make-column-dependencies-optional
branch
from
d035115 to
e8d6efb
Compare
rexledesma
force-pushed
the
rl/column-lineage-metadata
branch
from
a953dac to
11709ae
Compare
rexledesma
force-pushed
the
rl/make-column-dependencies-optional
branch
from
e8d6efb to
de86c55
Compare
erinkcochran87
approved these changes
Mar 19, 2024
rexledesma
force-pushed
the
rl/column-lineage-metadata
branch
from
11709ae to
4f0105d
Compare
rexledesma
force-pushed
the
rl/make-column-dependencies-optional
branch
from
de86c55 to
926c496
Compare
rexledesma
force-pushed
the
rl/column-lineage-metadata
branch
from
4f0105d to
0456137
Compare
rexledesma
force-pushed
the
rl/make-column-dependencies-optional
branch
from
926c496 to
a683924
Compare
|
Deploy preview for dagit-core-storybook ready! ✅ Preview Built with commit a683924. |
rexledesma
force-pushed
the
rl/column-lineage-metadata
branch
from
0456137 to
ba083cc
Compare
rexledesma
force-pushed
the
rl/make-column-dependencies-optional
branch
from
a683924 to
566fe88
Compare
rexledesma
force-pushed
the
rl/column-lineage-metadata
branch
from
ba083cc to
0628a9c
Compare
rexledesma
force-pushed
the
rl/make-column-dependencies-optional
branch
from
566fe88 to
c05e69a
Compare
Merge activity
|
rexledesma
force-pushed
the
rl/make-column-dependencies-optional
branch
from
c05e69a to
0024112
Compare
benpankow
pushed a commit
that referenced
this pull request
Mar 21, 2024
…20548) ## Summary & Motivation Allow the collection of column schema metadata for a dbt node's parents to be enabled or disabled. Disabling this collection prevents column dependencies from being emitted in the dbt node's materialization. ## How I Tested These Changes pytest, added docs
PedramNavid
pushed a commit
that referenced
this pull request
Mar 28, 2024
…20548) ## Summary & Motivation Allow the collection of column schema metadata for a dbt node's parents to be enabled or disabled. Disabling this collection prevents column dependencies from being emitted in the dbt node's materialization. ## How I Tested These Changes pytest, added docs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Allow the collection of column schema metadata for a dbt node's parents to be enabled or disabled. Disabling this collection prevents column dependencies from being emitted in the dbt node's materialization.
How I Tested These Changes
pytest, added docs