Batch fetching latest asset check executions #17650
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Current dependencies on/for this PR: This stack of pull requests is managed by Graphite. |
johannkm
commented
Nov 2, 2023
| @@ -138,3 +149,113 @@ def get_checks_for_asset( | |||
| ) | |||
|
|
|||
| return self._checks[asset_key] | |||
|
|
|||
|
|
|||
| def _execution_targets_latest_materialization( | |||
There was a problem hiding this comment.
moved over from fetch_asset_checks.py
This was referenced Nov 2, 2023
johannkm
force-pushed
the
johann/11-02-Batch_fetching_latest_asset_check_executions
branch
from
1f36e09 to
be4d6f7
Compare
johannkm
force-pushed
the
johann/11-02-remove_top_level_assetChecksOrError_resolver
branch
from
869b085 to
efa6d08
Compare
johannkm
force-pushed
the
johann/11-02-Batch_fetching_latest_asset_check_executions
branch
from
be4d6f7 to
aeaa9cf
Compare
Base automatically changed from
johann/11-02-remove_top_level_assetChecksOrError_resolver
to
master
November 2, 2023 20:58
johannkm
force-pushed
the
johann/11-02-Batch_fetching_latest_asset_check_executions
branch
from
aeaa9cf to
cdccbb8
Compare
rexledesma
approved these changes
Nov 2, 2023
| ): | ||
| self._asset_check = asset_check | ||
| self._can_execute_individually = can_execute_individually | ||
| self._exeuction_loader = execution_loader |
There was a problem hiding this comment.
typo
Suggested change
| self._exeuction_loader = execution_loader | |
| self._execution_loader = execution_loader |
| asset_graph = ExternalAssetGraph.from_workspace(self._context) | ||
| if limit_per_asset: | ||
| for asset_key, external_checks_for_asset in external_checks.items(): | ||
| external_checks[asset_key] = external_checks_for_asset[:limit_per_asset] |
There was a problem hiding this comment.
external_checks should be renamed external_checks_by_asset_key. It is currently confusing to parse since we iterate on it multiple times.
Suggested change
| external_checks[asset_key] = external_checks_for_asset[:limit_per_asset] | |
| external_checks_by_asset_key[asset_key] = external_checks_for_asset[:limit_per_asset] |
johannkm
force-pushed
the
johann/11-02-Batch_fetching_latest_asset_check_executions
branch
from
cdccbb8 to
0c8c36b
Compare
johannkm
deleted the
johann/11-02-Batch_fetching_latest_asset_check_executions
branch
|
Deploy preview for dagster-docs ready! Preview available at https://dagster-docs-nbfbpw3op-elementl.vercel.app Direct link to changed pages: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a batch loader for the executionForLatestMaterialization resolver, which we call to display the asset graph. Uses the event log method
get_latest_asset_check_execution_by_keyinstead of fetching each execution separately.There's still optimization to be done, e.g. we're still fetching the asset record once per check. I'll follow up with that in a separate diff.