Extract InOutMapper to begin refactoring AssetsDefinition construction process #22222
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This stack of pull requests is managed by Graphite. Learn more about stacking. |
This was referenced Jun 2, 2024
schrockn
force-pushed
the
aget-multi-asset-refactor-1
branch
from
e23b6f8 to
b2757bf
Compare
This was referenced Jun 2, 2024
schrockn
changed the title
sryza
previously requested changes
Jun 3, 2024
python_modules/dagster/dagster/_core/definitions/decorators/assets_definition_factory.py
Show resolved
Hide resolved
python_modules/dagster/dagster/_core/definitions/decorators/assets_definition_factory.py
Show resolved
Hide resolved
python_modules/dagster/dagster/_core/definitions/decorators/assets_definition_factory.py
Show resolved
Hide resolved
sryza
suggested changes
Jun 3, 2024
python_modules/dagster/dagster/_core/definitions/decorators/assets_definition_factory.py
Show resolved
Hide resolved
python_modules/dagster/dagster/_core/definitions/decorators/assets_definition_factory.py
Show resolved
Hide resolved
python_modules/dagster/dagster/_core/definitions/decorators/assets_definition_factory.py
Show resolved
Hide resolved
schrockn
force-pushed
the
aget-multi-asset-refactor-1
branch
from
b2757bf to
2eede8f
Compare
This was referenced Jun 8, 2024
Closed
Merged
Merged
salazarm
pushed a commit
that referenced
this pull request
Jun 10, 2024
…n process (#22222) ## Summary & Motivation `multi_asset` is a beast, as well as the entire `AssetsDefinition` creation machinery. This class introduces a class `InOutMapper` (renamed to `AssetsDefinitionFactory` upstack once its mandate increased) that begins to tease apart `multi_asset` which is nearly 300 LoC, has 37 local variables, and a huge dynamically scoped inner function. To see where this is going I have created a digest [PR](#22238) that demonstrate the before after. This also sets us up to consolidate the `AssetsDefinition` creation code paths, as it contains tons of duplicated code strewn about. Instead we will be able to invoke the decomposed code in the new factory functions. This was motivated by the discussion in #22221 that suggested we move a propose class to be within the inheritance hierarchy of `AssetsDefinition`. The complexity of logic surrounding the construction of said object is completely out of control, and I found it effectively intractable to do an inheritance scheme cleanly. ## How I Tested These Changes BK
This was referenced Jun 10, 2024
danielgafni
pushed a commit
to danielgafni/dagster
that referenced
this pull request
Jun 18, 2024
…n process (dagster-io#22222) ## Summary & Motivation `multi_asset` is a beast, as well as the entire `AssetsDefinition` creation machinery. This class introduces a class `InOutMapper` (renamed to `AssetsDefinitionFactory` upstack once its mandate increased) that begins to tease apart `multi_asset` which is nearly 300 LoC, has 37 local variables, and a huge dynamically scoped inner function. To see where this is going I have created a digest [PR](dagster-io#22238) that demonstrate the before after. This also sets us up to consolidate the `AssetsDefinition` creation code paths, as it contains tons of duplicated code strewn about. Instead we will be able to invoke the decomposed code in the new factory functions. This was motivated by the discussion in dagster-io#22221 that suggested we move a propose class to be within the inheritance hierarchy of `AssetsDefinition`. The complexity of logic surrounding the construction of said object is completely out of control, and I found it effectively intractable to do an inheritance scheme cleanly. ## How I Tested These Changes BK
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
multi_assetis a beast, as well as the entireAssetsDefinitioncreation machinery.This class introduces a class
InOutMapper(renamed toAssetsDefinitionFactoryupstack once its mandate increased) that begins to tease apartmulti_assetwhich is nearly 300 LoC, has 37 local variables, and a huge dynamically scoped inner function.To see where this is going I have created a digest PR that demonstrate the before after.
This also sets us up to consolidate the
AssetsDefinitioncreation code paths, as it contains tons of duplicated code strewn about. Instead we will be able to invoke the decomposed code in the new factory functions.This was motivated by the discussion in #22221 that suggested we move a propose class to be within the inheritance hierarchy of
AssetsDefinition. The complexity of logic surrounding the construction of said object is completely out of control, and I found it effectively intractable to do an inheritance scheme cleanly.How I Tested These Changes
BK