Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

select asset checks with build_dbt_asset_selection #18098

Conversation

johannkm
Copy link
Contributor

@johannkm johannkm commented Nov 17, 2023

Support things like

asset_selection = build_dbt_asset_selection([my_dbt_assets], dbt_select="test_name:not_null")

Previously, the selection would include any asset checks that targeted whatever dbt assets were selected. Now we rely on dbt to do that selection for us. Since we're hard coded to eager it's usually pretty similar, though there's a difference with relationship tests that I point out in a comment.

Another niche change is that previously this would select a non-dbt AssetCheck that targeted a selected dbt asset. I think that's a welcome change.

Both are minor breaking changes to an experimental feature that I don't think anyone will hit.

Closes #17372

@johannkm johannkm force-pushed the johann/11-16-select_asset_checks_with_build_dbt_asset_selection branch from 95da572 to d898be4 Compare November 17, 2023 04:17
@johannkm
Copy link
Contributor Author

Current dependencies on/for this PR:

This stack of pull requests is managed by Graphite.

@johannkm johannkm force-pushed the johann/11-16-select_asset_checks_with_build_dbt_asset_selection branch from d898be4 to 9b47619 Compare November 17, 2023 04:18
def test_selection_customers():
asset_selection = build_dbt_asset_selection([my_dbt_assets], dbt_select="customers")
assert asset_selection.resolve(asset_graph) == {AssetKey(["customers"])}
# all tests that reference model customers- includes a relationship test on orders
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little surprised by this, am I interpreting correctly?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default mode is eager, so this matches that behavior. Perhaps we should augment build_dbt_asset_selection to include indirect_selection as a parameter?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah seems like a likely request in the future

@johannkm johannkm requested review from sryza and rexledesma November 17, 2023 04:24
Copy link
Contributor

@rexledesma rexledesma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine, just some comments on readability

}


def test_all():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason why we're not parametrizing this? All the tests essentially have the same assertion.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO repeating a couple assert lines is easier to read and adjust than parameterizing test input, output, and name. Parameterizing makes sense for massive suites like AMP but I like keeping it simple when we can

@johannkm johannkm force-pushed the johann/11-16-select_asset_checks_with_build_dbt_asset_selection branch from dbf3745 to 610e9fa Compare November 27, 2023 17:47
@johannkm johannkm requested a review from rexledesma November 27, 2023 17:47
Copy link

Deploy preview for dagster-university ready!

✅ Preview
https://dagster-university-7tls8ilbp-elementl.vercel.app
https://johann-11-16-select-asset-checks-with-build-dbt-asset-selection.dagster-university.dagster-docs.io

Built with commit 610e9fa.
This pull request is being automatically deployed with vercel-action

Copy link

Deploy preview for dagit-core-storybook ready!

✅ Preview
https://dagit-core-storybook-nsx2lhdqv-elementl.vercel.app
https://johann-11-16-select-asset-checks-with-build-dbt-asset-selection.core-storybook.dagster-docs.io

Built with commit 610e9fa.
This pull request is being automatically deployed with vercel-action

Copy link
Contributor

@rexledesma rexledesma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might need to document a good sensible default for what dbt indirect selection we're using. A follow up PR for this + docs would be great.

@johannkm johannkm merged commit 14499e2 into master Nov 27, 2023
2 checks passed
@johannkm johannkm deleted the johann/11-16-select_asset_checks_with_build_dbt_asset_selection branch November 27, 2023 18:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

dbt asset checks select based on tags
2 participants