[external-assets] Build base asset jobs using AssetGraph #20227
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Mar 3, 2024
Merged
This was referenced Mar 3, 2024
smackesey
force-pushed
the
sean/allow-mixed-asset-jobs
branch
from
a3983e7 to
8ecfe36
Compare
smackesey
force-pushed
the
sean/external-assets-build-asset-jobs-with-asset-graph
branch
2 times, most recently
from
2f5ded7 to
f5a18c0
Compare
smackesey
force-pushed
the
sean/allow-mixed-asset-jobs
branch
from
8ecfe36 to
61f2d54
Compare
smackesey
force-pushed
the
sean/external-assets-build-asset-jobs-with-asset-graph
branch
2 times, most recently
from
6e9f0d2 to
3320908
Compare
smackesey
force-pushed
the
sean/allow-mixed-asset-jobs
branch
from
61f2d54 to
7da9c0f
Compare
smackesey
force-pushed
the
sean/external-assets-build-asset-jobs-with-asset-graph
branch
from
3320908 to
0faa6b0
Compare
smackesey
force-pushed
the
sean/allow-mixed-asset-jobs
branch
from
7da9c0f to
a636256
Compare
smackesey
force-pushed
the
sean/external-assets-build-asset-jobs-with-asset-graph
branch
from
0faa6b0 to
6a08c42
Compare
smackesey
force-pushed
the
sean/allow-mixed-asset-jobs
branch
from
a636256 to
d61c5bf
Compare
smackesey
force-pushed
the
sean/external-assets-build-asset-jobs-with-asset-graph
branch
from
6a08c42 to
7ff96a5
Compare
smackesey
force-pushed
the
sean/allow-mixed-asset-jobs
branch
2 times, most recently
from
77b327a to
4d232d2
Compare
smackesey
force-pushed
the
sean/external-assets-build-asset-jobs-with-asset-graph
branch
from
7ff96a5 to
0b9554c
Compare
smackesey
force-pushed
the
sean/allow-mixed-asset-jobs
branch
from
4d232d2 to
87d4d8f
Compare
smackesey
force-pushed
the
sean/external-assets-build-asset-jobs-with-asset-graph
branch
from
0b9554c to
e953bd4
Compare
smackesey
force-pushed
the
sean/allow-mixed-asset-jobs
branch
from
87d4d8f to
d0aaa70
Compare
smackesey
force-pushed
the
sean/external-assets-build-asset-jobs-with-asset-graph
branch
from
e953bd4 to
f54d8d2
Compare
smackesey
force-pushed
the
sean/allow-mixed-asset-jobs
branch
from
ec160de to
9b18924
Compare
smackesey
force-pushed
the
sean/external-assets-build-asset-jobs-with-asset-graph
branch
from
f2fca55 to
a92823f
Compare
smackesey
force-pushed
the
sean/allow-mixed-asset-jobs
branch
from
9b18924 to
86ebf75
Compare
smackesey
force-pushed
the
sean/external-assets-build-asset-jobs-with-asset-graph
branch
from
a92823f to
ae24e81
Compare
smackesey
force-pushed
the
sean/allow-mixed-asset-jobs
branch
from
86ebf75 to
06e3dba
Compare
smackesey
force-pushed
the
sean/external-assets-build-asset-jobs-with-asset-graph
branch
2 times, most recently
from
c89e7a3 to
aa172c5
Compare
smackesey
changed the base branch from
sean/allow-mixed-asset-jobs
to
sean/external-assets-asset-checks
smackesey
force-pushed
the
sean/external-assets-build-asset-jobs-with-asset-graph
branch
from
aa172c5 to
388323d
Compare
smackesey
force-pushed
the
sean/external-assets-asset-checks
branch
from
96c8dd2 to
aca467d
Compare
smackesey
force-pushed
the
sean/external-assets-build-asset-jobs-with-asset-graph
branch
from
388323d to
12501b8
Compare
smackesey
commented
Mar 8, 2024
| def res_upstream(context): | ||
| return context.resources.foo | ||
|
|
||
| @asset(required_resource_keys={"foo"}) | ||
| @asset |
There was a problem hiding this comment.
Tests using this were looking for res_midstream to be flagged specifically in the error message, when the same error would be thrown for all res_upstream, res_downstream as well. The reason the test did not previously fail is that res_midstream was incidentally being processed first, so that's where the error was.
The adjustment to base asset job creation in this PR causes res_midstream to no longer be processed first (which does not matter), but broke this test. Therefore I removed the required_resource_keys on res_upstream/res_downstream, as this fixed the failing test and none of the other tests using these assets actually depends on res_upstream/res_downstream having these req'd keys.
smackesey
force-pushed
the
sean/external-assets-asset-checks
branch
from
aca467d to
7408446
Compare
smackesey
force-pushed
the
sean/external-assets-build-asset-jobs-with-asset-graph
branch
from
12501b8 to
8ce31aa
Compare
smackesey
force-pushed
the
sean/external-assets-asset-checks
branch
from
7408446 to
f104d31
Compare
smackesey
force-pushed
the
sean/external-assets-build-asset-jobs-with-asset-graph
branch
from
8ce31aa to
0138965
Compare
smackesey
force-pushed
the
sean/external-assets-asset-checks
branch
from
f104d31 to
1c30a6b
Compare
smackesey
force-pushed
the
sean/external-assets-build-asset-jobs-with-asset-graph
branch
from
0138965 to
bff9f3d
Compare
smackesey
force-pushed
the
sean/external-assets-build-asset-jobs-with-asset-graph
branch
from
bff9f3d to
6674779
Compare
smackesey
deleted the
sean/external-assets-build-asset-jobs-with-asset-graph
branch
PedramNavid
pushed a commit
that referenced
this pull request
Mar 28, 2024
## Summary & Motivation Reap some benefits of the new `AssetGraph`. Construct the base jobs for a repository from the asset graph instead of lists of assets and checks. ## How I Tested These Changes Existing test suite.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Reap some benefits of the new
AssetGraph. Construct the base jobs for a repository from the asset graph instead of lists of assets and checks.How I Tested These Changes
Existing test suite.