fix after merge

docs/docs-next/docs/guides/data-assets.md

@@ -5,7 +5,3 @@ title: "Data assets"
 # Data assets
 
 TODO: fill in this section
-
-## Guides
-
-- [Passing data between assets](./passing-data-assets/passing-data-between-assets.md)

docs/docs-next/docs/guides/data-assets/adding-metadata-to-assets.md

@@ -1,6 +1,6 @@
 ---
 title: "Adding metadata to assets"
-sidebar_position: 4
+sidebar_position: 40
 sidebar_label: "Adding metadata"
 ---
 

docs/docs-next/docs/guides/data-assets/creating-asset-factories.md

@@ -1,6 +1,6 @@
 ---
 title: "Creating asset factories"
-sidebar_position: 5
+sidebar_position: 50
 sidebar_label: "Creating asset factories"
 ---
 

docs/docs-next/docs/guides/data-assets/creating-data-assets.md

@@ -1,6 +1,6 @@
 ---
 title: "Creating data assets"
-sidebar_position: 1
+sidebar_position: 10
 sidebar_label: "Creating data assets"
 ---
 

docs/docs-next/docs/guides/data-assets/creating-dependencies-between-assets.md

@@ -1,6 +1,6 @@
 ---
 title: "Creating dependencies between assets"
-sidebar_position: 2
+sidebar_position: 20
 sidebar_label: "Creating asset dependencies"
 ---
 

docs/docs-next/docs/guides/data-assets/passing-data-between-assets.md

@@ -1,7 +1,124 @@
 ---
-title: "Passing data between assets"
-sidebar_position: 3
+title: How to Pass Data Between Assets
+description: Learn how to pass data between assets in Dagster
+sidebar_position: 30
 sidebar_label: "Passing data between assets"
+last_update: 
+    date: 2024-08-11
+    author: Pedram Navid
 ---
 
-# Passing data between assets
+As you develop your data pipeline, you'll likely need to pass data between assets. By the end of this guide, you'll have a solid understanding of the different approaches to passing data between assets and when to use each one.
+
+---
+
+<details>
+  <summary>Prerequisites</summary>
+
+To follow the steps in this guide, you'll need:
+
+- A basic understanding of Dagster concepts such as assets and resources
+- Dagster installed, as well as the `dagster-duckdb-pandas` package
+</details>
+
+---
+
+## Overview
+
+In Dagster, assets are the building blocks of your data pipeline and it's common to want to pass data between them. This guide will help you understand how to pass data between assets. 
+
+There are three ways of passing data between assets:
+
+- Explicitly managing data, by using external storage
+- Implicitly managing data, using IO Managers
+- Avoiding passing data between assets altogether by combining several tasks into a single asset
+
+This guide walks through all three methods.
+
+---
+
+## Move Data Between Assets Explicitly Using External Storage
+
+A common and recommended approach to passing data between assets is explicitly managing data using external storage. This example pipeline uses a SQLite database as external storage:
+
+<CodeExample filePath="guides/data-assets/passing-data-assets/passing-data-explicit.py" language="python" title="Using External Storage" />
+
+In this example, the first asset opens a connection to the SQLite database and writes data to it. The second asset opens a connection to the same database and reads data from it.
+
+The benefits of this approach are:
+- It's explicit and easy to understand how data is stored and retrieved
+- You have maximum flexibility in terms of how and where data is stored, for example, based on environment
+
+The downsides of this approach are:
+- You need to manage connections and transactions manually
+- You need to handle errors and edge cases, for example, if the database is down or if a connection is closed
+
+## Move Data Between Assets Implicitly Using IO Managers
+
+Dagster's IO Managers are a powerful feature that manages data between assets by defining how data is read from and written to external storage. They help separate business logic from I/O operations, reducing boilerplate code and making it easier to change where data is stored. 
+
+I/O managers handle:
+1. **Input**: Reading data from storage and loading it into memory for use by dependent assets.
+2. **Output**: Writing data to the configured storage location.
+
+For a deeper understanding of IO Managers, check out the [Understanding IO Managers](/concepts/io-managers) guide.
+
+<CodeExample filePath="guides/data-assets/passing-data-assets/passing-data-io-manager.py" language="python" title="Using IO Managers" />
+
+In this example, a `DuckDBPandasIOManager` is instantiated to run using a local file. The IO manager handles both reading and writing to the database.
+
+:::warning
+
+This example works for local development, but in a production environment
+each step would execute in a separate environment and would not have access to the same file system. Consider a cloud-hosted environment for production purposes.
+
+:::
+
+The `people()` and `birds()` assets both write their dataframes to DuckDB 
+for persistent storage. The `combined_data()` asset requests data from both assets by adding them as parameters to the function, and the IO Manager handles the reading them from DuckDB and making them available to the `combined_data` function as dataframes.
+
+The benefits of this approach are:
+- The reading and writing of data is handled by the IO Manager, reducing boilerplate code
+- It's easy to swap out different IO Managers based on environments without changing the underlying asset computation
+
+The downsides of this approach are:
+- The IO Manager approach is less flexible should you need to customize how data is read or written to storage
+- Some decisions may be made by the IO Manager for you, such as naming conventions that can be hard to override.
+
+## Avoid Passing Data Between Assets by Combining Assets
+
+In some cases, you may find that you can avoid passing data between assets by
+carefully considering how you have modeled your pipeline:
+
+Consider this example:
+
+<CodeExample filePath="guides/data-assets/passing-data-assets/passing-data-avoid.py" language="python" title="Avoid Passing Data Between Assets" />
+
+This example downloads a zip file from Google Drive, unzips it, and loads the data into a pandas DataFrame. It relies on each asset running on the same file system to perform these operations. 
+
+The assets are modeled as tasks, rather than as data assets. For more information on the difference between tasks and data assets, check out the [Thinking in Assets](/concepts/thinking-in-assets) guide.
+
+In this refactor, the `download_files`, `unzip_files`, and `load_data` assets are combined into a single asset, `my_dataset`. This asset downloads the files, unzips them, and loads the data into a data warehouse.
+
+<CodeExample filePath="guides/data-assets/passing-data-assets/passing-data-rewrite-assets.py" language="python" title="Avoid Passing Data Between Assets" />
+
+This approach still handles passing data explicitly, but no longer does it across assets,
+instead within a single asset. This pipeline still assumes enough disk and
+memory available to handle the data, but for smaller datasets, it can work well.
+
+The benefits of this approach are:
+- All the computation that defines how an asset is created is contained within a single asset, making it easier to understand and maintain
+- It can be faster than relying on external storage, and doesn't require the overhead of setting up additional compute instances.
+
+
+The downsides of this approach are:
+- It makes certain assumptions about how much data is being processed
+- It can be difficult to reuse functions across assets, since they're tightly coupled to the data they produce
+- It may not always be possible to swap functionality based on the environment you are running in. For example, if you are running in a cloud environment, you may not have access to the local file system.
+
+
+---
+
+## Related Resources
+
+TODO: add links to relevant API documentation here.

docs/docs-next/docs/guides/data-assets/selecting-subsets-of-assets.md

@@ -1,6 +1,6 @@
 ---
 title: "Selecting subsets of assets"
-sidebar_position: 6
+sidebar_position: 60
 sidebar_label: "Selecting assets"
 ---