Skip to content

Commit

Permalink
page links
Browse files Browse the repository at this point in the history
  • Loading branch information
@C00ldudeNoonan
C00ldudeNoonan committed Dec 27, 2024
1 parent 1eb255c commit 4148df7
Show file tree
Hide file tree
Showing 7 changed files with 8 additions and 17 deletions.
2 changes: 1 addition & 1 deletion docs/docs-beta/docs/tutorial/etl-tutorial/01-etl-tutorial-introduction.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -133,4 +133,4 @@ followed by a bash code snippet for `dagster dev`

## Next steps

- Continue this tutorial by [creating and materializing assets](/tutorial/etl-tutorial/02-create-and-materialize-assets)
- Continue this tutorial by [creating and materializing assets](/tutorial/etl-tutorial/create-and-materialize-assets)
5 changes: 2 additions & 3 deletions docs/docs-beta/docs/tutorial/etl-tutorial/02-create-and-materialize-assets.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ last_update:
author: Alex Noonan
---

# Create and materialize assets

In the first step of the tutorial, you created your Dagster project with the raw data files. In this step, you will:
- Create your initial Definitions object
Expand All @@ -15,7 +14,7 @@ In the first step of the tutorial, you created your Dagster project with the raw

## 1. Create a Definitions object

In Dagster, the [Definitions](/api/definitions) object is where you define and organize various components within your project, such as assets and resources.
In Dagster, the [Definitions API docs](/todo) object is where you define and organize various components within your project, such as assets and resources.

Open the `definitions.py` file in the `etl_tutorial` directory and copy the following code into it:

Expand Down Expand Up @@ -105,4 +104,4 @@ To materialize your assets:

## Next steps

- Continue this tutorial with your with your [asset dependencies](/tutorial/etl-tutorial/03-creating-a-downstream-asset)
- Continue this tutorial with your with your [asset dependencies](/tutorial/etl-tutorial/creating-a-downstream-asset)
4 changes: 1 addition & 3 deletions ...beta/docs/tutorial/etl-tutorial/03-create-and-materialize-a-downstream-asset.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,6 @@ last_update:
author: Alex Noonan
---

# Asset Dependencies

Now that we have the raw data loaded into DuckDB, we need to create a [downstream asset](/guides/build/assets-concepts/asset-dependencies) that combines the upstream assets together. In this step, you will:

- Create a downstream asset
Expand Down Expand Up @@ -43,4 +41,4 @@ Your Definitions object should now look like this:

## Next steps

- Continue this tutorial with [create and materialize a partitioned asset](/tutorial/etl-tutorial/05-ensuring-data-quality-with-asset-checks)
- Continue this tutorial with [create and materialize a partitioned asset](/tutorial/etl-tutorial/ensuring-data-quality-with-asset-checks)
4 changes: 1 addition & 3 deletions ...s-beta/docs/tutorial/etl-tutorial/04-ensuring-data-quality-with-asset-checks.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,6 @@ last_update:
author: Alex Noonan
---

# Asset checks

Data Quality is critical in data pipelines. Much like in a factory producing cars, inspecting parts after they complete certain steps ensures that defects are caught before the car is completely assembled.

Check warning on line 8 in docs/docs-beta/docs/tutorial/etl-tutorial/04-ensuring-data-quality-with-asset-checks.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Dagster.chars-eol-whitespace] Remove whitespace characters from the end of the line.

Raw Output:
{"message": "[Dagster.chars-eol-whitespace] Remove whitespace characters from the end of the line.", "location": {"path": "docs/docs-beta/docs/tutorial/etl-tutorial/04-ensuring-data-quality-with-asset-checks.md", "range": {"start": {"line": 8, "column": 206}}}, "severity": "WARNING"}

In Dagster, you define [asset checks](/guides/test/asset-checks) in a similar way that you would define an Asset. In this step you will:
Expand Down Expand Up @@ -50,4 +48,4 @@ Asset checks will run when an asset is materialized, but asset checks can also b

## Next steps

- Continue this tutorial with [Asset Checks](/tutorial/etl-tutorial/04-ensuring-data-quality-with-asset-checks)
- Continue this tutorial with [Asset Checks](/tutorial/etl-tutorial/create-and-materializa-partitioned-asset)
4 changes: 1 addition & 3 deletions ...-beta/docs/tutorial/etl-tutorial/05-create-and-materialize-partitioned-asset.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,6 @@ last_update:
author: Alex Noonan
---

# Partitions

[Partitions](/guides/create-a-pipeline/partitioning) are a core abstraction in Dagster, they are how you manage large datasets, process incremental updates, and improve pipeline performance. In Dagster you can partition assets the following ways:

1. Time-based: Split data by time periods (e.g., daily, monthly)

Check warning on line 11 in docs/docs-beta/docs/tutorial/etl-tutorial/05-create-and-materialize-partitioned-asset.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Dagster.latin] Use 'for example' instead of 'e.g.', but consider rewriting the sentence.

Raw Output:
{"message": "[Dagster.latin] Use 'for example' instead of 'e.g.', but consider rewriting the sentence.", "location": {"path": "docs/docs-beta/docs/tutorial/etl-tutorial/05-create-and-materialize-partitioned-asset.md", "range": {"start": {"line": 11, "column": 44}}}, "severity": "WARNING"}
Expand Down Expand Up @@ -177,4 +175,4 @@ To materialize these assets :

## Next Steps

Now that we have the main assets in our ETL pipeline, its time to add [automation to our pipeline](/tutorial/etl-tutorial/06-automating-your-pipeline)
Now that we have the main assets in our ETL pipeline, its time to add [automation to our pipeline](/tutorial/etl-tutorial/automating-your-pipeline)
4 changes: 1 addition & 3 deletions docs/docs-beta/docs/tutorial/etl-tutorial/06-automating-your-pipeline.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,6 @@ last_update:
author: Alex Noonan
---

# Automation

There are several ways to automate pipelines and assets [in Dagster](/guides/automate).

In this step you will:
Expand Down Expand Up @@ -54,4 +52,4 @@ Additionally if you navigate to the runs tab you will see that materializations

## Next steps

- Continue this tutorial with adding a [sensor based asset](/tutorial/etl-tutorial/07-creating-a-sensor-asset)
- Continue this tutorial with adding a [sensor based asset](/tutorial/etl-tutorial/creating-a-sensor-asset)
2 changes: 1 addition & 1 deletion docs/docs-beta/docs/tutorial/etl-tutorial/07-creating-a-sensor-asset.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -68,4 +68,4 @@ sensors include the following elements:

Now that we have our complete project, the next step is to refactor the project into more a more manageable structure so we can add to it as needed.

Finish the tutorial with [refactoring the project](/tutorial/etl-tutorial/08-refactoring-the-project)
Finish the tutorial with [refactoring the project](/tutorial/etl-tutorial/refactoring-the-project)

0 comments on commit 4148df7

Please sign in to comment.