Skip to content

Commit

Permalink
[1.8][external assets] remove external_assets_from_specs from conce…
Browse files Browse the repository at this point in the history
…pt docs (#23200)

## Summary & Motivation

Replaces them with direct usage of `AssetSpec`, which is enabled by
#22961.

## How I Tested These Changes
  • Loading branch information
sryza authored Aug 2, 2024
1 parent 34e898b commit 54dd213
Show file tree
Hide file tree
Showing 6 changed files with 20 additions and 40 deletions.
38 changes: 12 additions & 26 deletions docs/content/concepts/assets/external-assets.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,6 @@ An **external asset** is an asset that is visible in Dagster but executed by an

In this case, you could use an external asset to leverage Dagster's event log and tooling without using the orchestrator. This allows you to maintain data lineage, observability, and data quality without unnecessary migrations.

### What about Source Assets?

<PyObject object="SourceAsset" pluralize /> can be used to model data that's produced
by a process Dagster doesn't control, such as a daily file drop into Amazon S3.

External assets can accomplish this, and more. As a result, Source Assets will be replaced with external assets in the near future.

---

## Uses and limitations
Expand All @@ -38,26 +31,25 @@ The following aren't currently supported when using external assets:

## Relevant APIs

| Name | Description |
| ------------------------------------------------ | ---------------------------------------------------------------------------------------------- |
| <PyObject object="external_assets_from_specs" /> | Creates a list of <PyObject object="AssetsDefinition"/> objects that represent external assets |
| <PyObject object="AssetSpec" /> | An object that represents the metadata of a particular asset |
| Name | Description |
| ------------------------------- | ---------------------------------------------------------------------------------------------------- |
| <PyObject object="AssetSpec" /> | An object that represents the metadata of a particular asset without representing how it's computed. |

---

## Defining external assets

The following code declares a single external asset that represents a file in S3 and passes it to a <PyObject object="Definitions"/> object:
External assets are defined using the <PyObject object="AssetSpec" /> class. An <PyObject object="AssetSpec" /> represents metadata about an asset without describing how it's computed. The following code declares a single external asset that represents a file in S3 and passes it to a <PyObject object="Definitions"/> object:

<TabGroup>
<TabItem name="Asset definition">

Click the **Asset in the Dagster UI** tab to see how this asset would be rendered in the Dagster UI.

```python file=/concepts/assets/external_assets/single_declaration.py
from dagster import AssetSpec, Definitions, external_asset_from_spec
from dagster import AssetSpec, Definitions

defs = Definitions(assets=[external_asset_from_spec(AssetSpec("file_in_s3"))])
defs = Definitions(assets=[AssetSpec("file_in_s3")])
```

---
Expand Down Expand Up @@ -95,12 +87,12 @@ In the following example, we have two assets: `raw_logs` and `processed_logs`. T
Click the **Assets in the Dagster UI** tab to see how these assets would be rendered in the Dagster UI.

```python file=/concepts/assets/external_assets/external_asset_deps.py
from dagster import AssetSpec, Definitions, external_assets_from_specs
from dagster import AssetSpec, Definitions

raw_logs = AssetSpec("raw_logs")
processed_logs = AssetSpec("processed_logs", deps=[raw_logs])

defs = Definitions(assets=external_assets_from_specs([raw_logs, processed_logs]))
defs = Definitions(assets=[raw_logs, processed_logs])
```

---
Expand Down Expand Up @@ -134,7 +126,7 @@ Fully-managed assets can depend on external assets. In this example, the `aggreg
Click the **Assets in the Dagster UI** tab to see how these assets would be rendered in the Dagster UI.

```python file=/concepts/assets/external_assets/normal_asset_depending_on_external.py
from dagster import AssetSpec, Definitions, asset, external_assets_from_specs
from dagster import AssetSpec, Definitions, asset

raw_logs = AssetSpec("raw_logs")
processed_logs = AssetSpec("processed_logs", deps=[raw_logs])
Expand All @@ -146,9 +138,7 @@ def aggregated_logs() -> None:
...


defs = Definitions(
assets=[aggregated_logs, *external_assets_from_specs([raw_logs, processed_logs])]
)
defs = Definitions(assets=[aggregated_logs, raw_logs, processed_logs])
```

</TabItem>
Expand Down Expand Up @@ -196,7 +186,6 @@ from dagster import (
Definitions,
SensorEvaluationContext,
SensorResult,
external_asset_from_spec,
sensor,
)

Expand All @@ -221,7 +210,7 @@ def keep_external_asset_a_up_to_date(context: SensorEvaluationContext) -> Sensor


defs = Definitions(
assets=[external_asset_from_spec(AssetSpec("external_asset_a"))],
assets=[AssetSpec("external_asset_a")],
sensors=[keep_external_asset_a_up_to_date],
)
```
Expand Down Expand Up @@ -253,7 +242,6 @@ from dagster import (
AssetSpec,
Definitions,
OpExecutionContext,
external_asset_from_spec,
job,
op,
)
Expand All @@ -269,9 +257,7 @@ def a_job() -> None:
an_op()


defs = Definitions(
assets=[external_asset_from_spec(AssetSpec("external_asset"))], jobs=[a_job]
)
defs = Definitions(assets=[AssetSpec("external_asset")], jobs=[a_job])
```

---
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from dagster import AssetSpec, Definitions, external_assets_from_specs
from dagster import AssetSpec, Definitions

raw_logs = AssetSpec("raw_logs")
processed_logs = AssetSpec("processed_logs", deps=[raw_logs])

defs = Definitions(assets=external_assets_from_specs([raw_logs, processed_logs]))
defs = Definitions(assets=[raw_logs, processed_logs])
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
Definitions,
SensorEvaluationContext,
SensorResult,
external_asset_from_spec,
sensor,
)

Expand All @@ -31,6 +30,6 @@ def keep_external_asset_a_up_to_date(context: SensorEvaluationContext) -> Sensor


defs = Definitions(
assets=[external_asset_from_spec(AssetSpec("external_asset_a"))],
assets=[AssetSpec("external_asset_a")],
sensors=[keep_external_asset_a_up_to_date],
)
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from dagster import AssetSpec, Definitions, asset, external_assets_from_specs
from dagster import AssetSpec, Definitions, asset

raw_logs = AssetSpec("raw_logs")
processed_logs = AssetSpec("processed_logs", deps=[raw_logs])
Expand All @@ -10,6 +10,4 @@ def aggregated_logs() -> None:
...


defs = Definitions(
assets=[aggregated_logs, *external_assets_from_specs([raw_logs, processed_logs])]
)
defs = Definitions(assets=[aggregated_logs, raw_logs, processed_logs])
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
from dagster import AssetSpec, Definitions, external_asset_from_spec
from dagster import AssetSpec, Definitions

defs = Definitions(assets=[external_asset_from_spec(AssetSpec("file_in_s3"))])
defs = Definitions(assets=[AssetSpec("file_in_s3")])
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
AssetSpec,
Definitions,
OpExecutionContext,
external_asset_from_spec,
job,
op,
)
Expand All @@ -19,6 +18,4 @@ def a_job() -> None:
an_op()


defs = Definitions(
assets=[external_asset_from_spec(AssetSpec("external_asset"))], jobs=[a_job]
)
defs = Definitions(assets=[AssetSpec("external_asset")], jobs=[a_job])

1 comment on commit 54dd213

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs ready!

✅ Preview
https://dagster-docs-kakaouboh-elementl.vercel.app
https://master.dagster.dagster-docs.io

Built with commit 54dd213.
This pull request is being automatically deployed with vercel-action

Please sign in to comment.