Skip to content

Commit

Permalink
[docs] Add guide for code references
Browse files Browse the repository at this point in the history
  • Loading branch information
benpankow committed Jul 2, 2024
1 parent c0c95bf commit d833a1c
Show file tree
Hide file tree
Showing 12 changed files with 163 additions and 4 deletions.
8 changes: 4 additions & 4 deletions docs/content/_navigation.json
Original file line number Diff line number Diff line change
Expand Up @@ -1200,13 +1200,13 @@
{
"title": "Experimental features",
"children": [
{
"title": "Airbyte ingestion as code",
"path": "/guides/dagster/airbyte-ingestion-as-code"
},
{
"title": "Asset versioning and caching",
"path": "/guides/dagster/asset-versioning-and-caching"
},
{
"title": "Linking to code in Dagster with code references",
"path": "/guides/dagster/code-references"
}
]
}
Expand Down
Binary file modified docs/content/api/modules.json.gz
Binary file not shown.
Binary file modified docs/content/api/searchindex.json.gz
Binary file not shown.
Binary file modified docs/content/api/sections.json.gz
Binary file not shown.
5 changes: 5 additions & 0 deletions docs/content/concepts/metadata-tags/asset-metadata.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ Dagster supports attaching a few different types of definition metadata:
- [**Arbitrary metadata**](#arbitrary-metadata-using-the-metadata-parameter), such as the storage location of the table produced by the asset
- [**Asset owners**](#asset-owners), which are the people and/or teams who own the asset
- [**Table and column metadata**](#table-and-column-metadata), which provides additional context about a tabular asset, such as its schema or row count
- [**Code references**](#code-references), which link to the source code of the asset locally or in your source control repository

### Arbitrary metadata using the metadata parameter

Expand Down Expand Up @@ -83,6 +84,10 @@ from dagster import asset
def leads(): ...
```

### Code references

Attaching code references to an asset definition allows you to easily navigate to the asset's source code, either locally in your editor or in your source control repository. For more information, refer to the [Code references guide](/guides/dagster/code-references).

---

## Attaching materialization metadata
Expand Down
2 changes: 2 additions & 0 deletions docs/content/guides.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -78,4 +78,6 @@ Learn to apply [Dagster concepts](/concepts) to your work, explore experimental

- [Asset versioning and caching](/guides/dagster/asset-versioning-and-caching) - Memoize assets using Dagster's data versioning system

- [Linking to assets' code with code references](/guides/dagster/code-references) - Attach code references to your Dagster assets to easily navigate to the code that backs the asset

[def]: /guides/dagster/
109 changes: 109 additions & 0 deletions docs/content/guides/dagster/code-references.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
---
title: Linking to assets' code with code references | Dagster Docs
---

# Linking to assets' code with code references

<Note>
This feature is considered <strong>experimental</strong> and is under active
development. This guide will be updated as we roll out new features.
</Note>

Attaching code reference metadata to your Dagster asset definitions allows you to easily view those assets' source code from the Dagster UI:

- **In local development**, navigate directly to the code backing an asset in your editor
- **In your production environment**, link to source code in your source control repository to see the full change history for an asset

In this guide, we'll show you how to automatically and manually attach code references to your Dagster assets.

---

## Prerequisites

To complete the steps in this guide, you'll need:

- A set of Dagster asset definitions that you want to link to code
- Dagster version `1.7.6` or newer

---

## Automatically attaching code references to asset definitions

To automatically attach code references to the asset's function definition, you can use the `dagster._core.definitions.metadata.with_source_code_references` utility. Any asset definitions passed to the utility will have their source file attached as metadata.

```python file=/guides/dagster/code_references/with_source_code_references.py
from dagster import Definitions, asset
from dagster._core.definitions.metadata import with_source_code_references


@asset
def my_asset(): ...


@asset
def another_asset(): ...


defs = Definitions(assets=with_source_code_references([my_asset, another_asset]))
```

A link to the asset's source in `with_source_code_references.py` will then be visible in the **Asset Catalog** view in the Dagster UI:

<center>
<Image
alt="Asset catalog view showing link to with_source_code_references.py"
src="/images/guides/code-references/with_source_code_references.png"
width={1590}
height={326}
/>
</center>

---

## Manually attaching code references to asset definitions

In some cases, you may want to manually attach code references to your asset definitions. Some assets may have a more complex source structure, such as an asset whose definition is spread across multiple Python source files or an asset which is partially defined with a `.sql` model file.

To manually attach code references to an asset definition, use `CodeReferencesMetadataValue`. You can then choose to augment these manual references with `with_source_code_references`:

```python file=/guides/dagster/code_references/manual_references.py
import os

from dagster import Definitions, asset
from dagster._core.definitions.metadata import (
CodeReferencesMetadataValue,
LocalFileCodeReference,
with_source_code_references,
)


@asset(
metadata={
"dagster/code_references": CodeReferencesMetadataValue(
code_references=[
LocalFileCodeReference(
file_path=os.path.join(os.path.dirname(__file__), "source.yaml"),
# Label and line number are optional
line_number=1,
label="Model YAML",
)
]
)
}
)
def my_asset_modeled_in_yaml(): ...


defs = Definitions(assets=with_source_code_references([my_asset_modeled_in_yaml]))
```

Each of the code references to `manual_references.py` will be visible in the **Asset details** page in the Dagster UI:

<center>
<Image
alt="Asset details view showing link to multiple files"
src="/images/guides/code-references/manual_references.png"
width={1592}
height={389}
/>
</center>
2 changes: 2 additions & 0 deletions docs/content/guides/experimental-features.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,5 @@ title: Advanced and experimental feature guides | Dagster Docs
## Experimental features

- [Asset versioning and caching](/guides/dagster/asset-versioning-and-caching) - Memoize assets using Dagster's data versioning system

- [Linking to assets' code with code references](/guides/dagster/code-references) - Attach code references to your Dagster assets to easily navigate to the code that backs the asset
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
import os

from dagster import Definitions, asset
from dagster._core.definitions.metadata import (
CodeReferencesMetadataValue,
LocalFileCodeReference,
with_source_code_references,
)


@asset(
metadata={
"dagster/code_references": CodeReferencesMetadataValue(
code_references=[
LocalFileCodeReference(
file_path=os.path.join(os.path.dirname(__file__), "source.yaml"),
# Label and line number are optional
line_number=1,
label="Model YAML",
)
]
)
}
)
def my_asset_modeled_in_yaml(): ...


defs = Definitions(assets=with_source_code_references([my_asset_modeled_in_yaml]))
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
from dagster import Definitions, asset
from dagster._core.definitions.metadata import with_source_code_references


@asset
def my_asset(): ...


@asset
def another_asset(): ...


defs = Definitions(assets=with_source_code_references([my_asset, another_asset]))

0 comments on commit d833a1c

Please sign in to comment.