-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[docs] Add guide for code references
- Loading branch information
Showing
12 changed files
with
163 additions
and
4 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
Binary file not shown.
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,109 @@ | ||
--- | ||
title: Linking to assets' code with code references | Dagster Docs | ||
--- | ||
|
||
# Linking to assets' code with code references | ||
|
||
<Note> | ||
This feature is considered <strong>experimental</strong> and is under active | ||
development. This guide will be updated as we roll out new features. | ||
</Note> | ||
|
||
Attaching code reference metadata to your Dagster asset definitions allows you to easily view those assets' source code from the Dagster UI: | ||
|
||
- **In local development**, navigate directly to the code backing an asset in your editor | ||
- **In your production environment**, link to source code in your source control repository to see the full change history for an asset | ||
|
||
In this guide, we'll show you how to automatically and manually attach code references to your Dagster assets. | ||
|
||
--- | ||
|
||
## Prerequisites | ||
|
||
To complete the steps in this guide, you'll need: | ||
|
||
- A set of Dagster asset definitions that you want to link to code | ||
- Dagster version `1.7.6` or newer | ||
|
||
--- | ||
|
||
## Automatically attaching code references to asset definitions | ||
|
||
To automatically attach code references to the asset's function definition, you can use the `dagster._core.definitions.metadata.with_source_code_references` utility. Any asset definitions passed to the utility will have their source file attached as metadata. | ||
|
||
```python file=/guides/dagster/code_references/with_source_code_references.py | ||
from dagster import Definitions, asset | ||
from dagster._core.definitions.metadata import with_source_code_references | ||
|
||
|
||
@asset | ||
def my_asset(): ... | ||
|
||
|
||
@asset | ||
def another_asset(): ... | ||
|
||
|
||
defs = Definitions(assets=with_source_code_references([my_asset, another_asset])) | ||
``` | ||
|
||
A link to the asset's source in `with_source_code_references.py` will then be visible in the **Asset Catalog** view in the Dagster UI: | ||
|
||
<center> | ||
<Image | ||
alt="Asset catalog view showing link to with_source_code_references.py" | ||
src="/images/guides/code-references/with_source_code_references.png" | ||
width={1590} | ||
height={326} | ||
/> | ||
</center> | ||
|
||
--- | ||
|
||
## Manually attaching code references to asset definitions | ||
|
||
In some cases, you may want to manually attach code references to your asset definitions. Some assets may have a more complex source structure, such as an asset whose definition is spread across multiple Python source files or an asset which is partially defined with a `.sql` model file. | ||
|
||
To manually attach code references to an asset definition, use `CodeReferencesMetadataValue`. You can then choose to augment these manual references with `with_source_code_references`: | ||
|
||
```python file=/guides/dagster/code_references/manual_references.py | ||
import os | ||
|
||
from dagster import Definitions, asset | ||
from dagster._core.definitions.metadata import ( | ||
CodeReferencesMetadataValue, | ||
LocalFileCodeReference, | ||
with_source_code_references, | ||
) | ||
|
||
|
||
@asset( | ||
metadata={ | ||
"dagster/code_references": CodeReferencesMetadataValue( | ||
code_references=[ | ||
LocalFileCodeReference( | ||
file_path=os.path.join(os.path.dirname(__file__), "source.yaml"), | ||
# Label and line number are optional | ||
line_number=1, | ||
label="Model YAML", | ||
) | ||
] | ||
) | ||
} | ||
) | ||
def my_asset_modeled_in_yaml(): ... | ||
|
||
|
||
defs = Definitions(assets=with_source_code_references([my_asset_modeled_in_yaml])) | ||
``` | ||
|
||
Each of the code references to `manual_references.py` will be visible in the **Asset details** page in the Dagster UI: | ||
|
||
<center> | ||
<Image | ||
alt="Asset details view showing link to multiple files" | ||
src="/images/guides/code-references/manual_references.png" | ||
width={1592} | ||
height={389} | ||
/> | ||
</center> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+128 KB
docs/next/public/images/guides/code-references/with_source_code_references.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
28 changes: 28 additions & 0 deletions
28
examples/docs_snippets/docs_snippets/guides/dagster/code_references/manual_references.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
import os | ||
|
||
from dagster import Definitions, asset | ||
from dagster._core.definitions.metadata import ( | ||
CodeReferencesMetadataValue, | ||
LocalFileCodeReference, | ||
with_source_code_references, | ||
) | ||
|
||
|
||
@asset( | ||
metadata={ | ||
"dagster/code_references": CodeReferencesMetadataValue( | ||
code_references=[ | ||
LocalFileCodeReference( | ||
file_path=os.path.join(os.path.dirname(__file__), "source.yaml"), | ||
# Label and line number are optional | ||
line_number=1, | ||
label="Model YAML", | ||
) | ||
] | ||
) | ||
} | ||
) | ||
def my_asset_modeled_in_yaml(): ... | ||
|
||
|
||
defs = Definitions(assets=with_source_code_references([my_asset_modeled_in_yaml])) |
13 changes: 13 additions & 0 deletions
13
...docs_snippets/docs_snippets/guides/dagster/code_references/with_source_code_references.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
from dagster import Definitions, asset | ||
from dagster._core.definitions.metadata import with_source_code_references | ||
|
||
|
||
@asset | ||
def my_asset(): ... | ||
|
||
|
||
@asset | ||
def another_asset(): ... | ||
|
||
|
||
defs = Definitions(assets=with_source_code_references([my_asset, another_asset])) |