[external assets] source assets -> external assets #18217
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @smackesey and the rest of your teammates on |
|
Deploy preview for dagit-core-storybook ready! ✅ Preview Built with commit 5ebeac7. |
smackesey
force-pushed
the
sean/convert-source-assets
branch
from
94579d5 to
76e746d
Compare
smackesey
force-pushed
the
sean/external-assets-observe-result
branch
from
5503f7f to
33d754c
Compare
smackesey
force-pushed
the
sean/convert-source-assets
branch
from
76e746d to
1aa3b00
Compare
smackesey
force-pushed
the
sean/external-assets-observe-result
branch
from
33d754c to
fc5f310
Compare
smackesey
force-pushed
the
sean/convert-source-assets
branch
from
1aa3b00 to
2efc378
Compare
This was referenced Feb 1, 2024
Merged
smackesey
force-pushed
the
sean/external-assets-observe-result
branch
from
fc5f310 to
375c613
Compare
smackesey
force-pushed
the
sean/convert-source-assets
branch
from
2efc378 to
1bfc23d
Compare
smackesey
force-pushed
the
sean/external-assets-observe-result
branch
4 times, most recently
from
ce7dd71 to
dc76212
Compare
smackesey
force-pushed
the
sean/convert-source-assets
branch
2 times, most recently
from
e3bb98c to
8b0fb54
Compare
smackesey
force-pushed
the
sean/external-assets-observe-result
branch
2 times, most recently
from
1625c87 to
1670c1f
Compare
Base automatically changed from
sean/external-assets-observe-result
to
master
February 5, 2024 15:21
smackesey
force-pushed
the
sean/convert-source-assets
branch
2 times, most recently
from
e3772e2 to
0fb2e11
Compare
smackesey
force-pushed
the
sean/convert-source-assets
branch
2 times, most recently
from
a8a9ee5 to
db14c71
Compare
smackesey
force-pushed
the
sean/convert-source-assets
branch
from
db14c71 to
f5c0284
Compare
smackesey
force-pushed
the
sean/convert-source-assets
branch
from
f5c0284 to
5ebeac7
Compare
smackesey
force-pushed
the
sean/allow-mixed-asset-jobs
branch
2 times, most recently
from
d881aa6 to
a1490ca
Compare
smackesey
force-pushed
the
sean/allow-mixed-asset-jobs
branch
2 times, most recently
from
36d76b2 to
be3a82e
Compare
smackesey
force-pushed
the
sean/allow-mixed-asset-jobs
branch
from
be3a82e to
33f844e
Compare
smackesey
changed the base branch from
sean/allow-mixed-asset-jobs
to
sean/remove-source-asset-from-jobs
smackesey
force-pushed
the
sean/convert-source-assets
branch
from
5ebeac7 to
24d608d
Compare
smackesey
force-pushed
the
sean/remove-source-asset-from-jobs
branch
6 times, most recently
from
3233518 to
08255d1
Compare
This was referenced Feb 12, 2024
smackesey
force-pushed
the
sean/remove-source-asset-from-jobs
branch
2 times, most recently
from
5b214ac to
3bfbef8
Compare
This was referenced Feb 13, 2024
Merged
smackesey
force-pushed
the
sean/remove-source-asset-from-jobs
branch
from
3bfbef8 to
05b414b
Compare
|
Superseded by another stack. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Internally convert source assets to external assets. This happens at the level of
RepositoryData-- we do the conversion here instead of inDefinitionsso that conversion works for both@repositoryandDefinitions."Source assets" no longer exist as a concept at the level of
ExternalAssetNode, but still exist over the GQL API to keep the representation in Dagster UI constant. A newisExternalfield is also added to GQL asset nodes.A caveat is that
auto_observe_interval_minutescan not be perfectly translated to an auto-materialize policy, because there is no way to capture "every N minutes" exactly as a cron expression (andAutoMaterializeRule.materialize_on_cronis the closest option available). I used a mapping that gets us pretty close though where we round to the nearest minute/hour/day for the number of minutes.How I Tested These Changes