diff --git a/docs/content/_navigation.json b/docs/content/_navigation.json index e73be8dac6006..499a50ec9f833 100644 --- a/docs/content/_navigation.json +++ b/docs/content/_navigation.json @@ -909,6 +909,10 @@ { "title": "Experimental features", "children": [ + { + "title": "Dagster Pipes", + "path": "/guides/dagster-pipes" + }, { "title": "Using custom run coordinators", "path": "/guides/dagster/run-attribution" diff --git a/docs/content/guides.mdx b/docs/content/guides.mdx index 21c33a0beaba6..e0c648a84a892 100644 --- a/docs/content/guides.mdx +++ b/docs/content/guides.mdx @@ -76,6 +76,12 @@ Learn to apply [Dagster concepts](/concepts) to your work, explore experimental ### Experimental features +#### Dagster Pipes + +- [Dagster Pipes](/guides/dagster-pipes) - A high-level look at Dagster Pipes, a toolkit for building integrations between Dagster and external execution environments + +#### Other features + - [Using Custom Run Coordinators to perform run attribution](/guides/dagster/run-attribution) - A look at using a Custom Run Coordinator to perform run attribution - [Airbyte ingestion as code](/guides/dagster/airbyte-ingestion-as-code) - Configure Airbyte connections with Dagster diff --git a/docs/content/guides/dagster-pipes.mdx b/docs/content/guides/dagster-pipes.mdx new file mode 100644 index 0000000000000..34e3843d72a93 --- /dev/null +++ b/docs/content/guides/dagster-pipes.mdx @@ -0,0 +1,71 @@ +--- +title: Dagster Pipes | Dagster Docs +description: "Dagster Pipes provides a protocol between the orchestration environment (Dagster) and external execution (ex: Databricks) and a toolkit for building implementations of that protocol." +--- + +# Dagster Pipes (Experimental) + + + This feature is currently experimental. + + +Dagster Pipes is a toolkit for building integrations between Dagster and external execution environments. It standardizes the process of passing parameters, injecting context information, ingesting logs, and collecting metadata all while remaining agnostic to how remote computations are launched in those environments. This enables the separation of orchestration and business logic in the Dagster ecosystem. + +It also smooths the process of incorporating pre-existing code, business logic, and execution environments into Dagster. With Dagster Pipes and a few lines of code, you can execute your code through the orchestrator. In turn, you can stream logs and metadata back to Dagster so you can leverage its observability, lineage, cataloging, and debugging capabilities. + +--- + +## Benefits + +With Dagster Pipes, you can: + +- Incorporate existing code into Dagster without huge refactors +- Onboard stakeholder teams onto Dagster incrementally +- Run code in external environments, and: + - Easily pass parameters to the code + - Stream unstructured logs and structured metadata back to Dagster +- Separate orchestration and business logic environments +- Use languages other than Python with Dagster + +--- + +## Limitations + +While Dagster Pipes is lightweight and flexible, there are a few limitations to be aware of: + +- **Step launchers and Pipes can't currently be used together.** Dagster Pipes is a lightweight alternative to step launchers. Think of this as an either-or situation - you can use step launchers **or** you can use Dagster Pipes. If your external code requires the core Dagster module (see below), you should use step launchers instead of Pipes. + +- **Some Dagster concepts aren't supported for use in external processes.** Dagster Pipes (`dagster-pipes`) doesn't include the core Dagster (`dagster`) library. As such, concepts like resources and I/O managers, which are included in `dagster`, aren't available for use in external processes executed by Dagster Pipes. + + For example, I/O managers aren't currently supported, as Dagster Pipes isn't built for processing data in memory. To process data in memory in Pyspark, you could use an I/O manager [as demonstrated in the Pyspark integration guide](/integrations/spark#running-pyspark-code-in-assets). Otherwise, in a remote case, you can use Pipes. + +--- + +## How it works + +Dagster Pipes provides a protocol between the orchestration environment (Dagster) and external execution (ex: Databricks) and a toolkit for building implementations of that protocol. + +When Dagster Pipes is invoked, several steps will be carried out in **Dagster's orchestration process** and in the **external process**, such as Databricks. + +### In the orchestration process (Dagster) + +When Dagster Pipes is called in a Dagster asset, Dagster launches the external process with parameters and context information (ex: `partition_key`, `asset_key`, etc.) + + + +### In the external process + +The process starts and loads the context info provided by Dagster. While the process runs, execution data, logs, and any specified metadata are streamed back to Dagster. + +After Dagster receives the data from the external process, it’ll be visible in the [Dagster UI](/concepts/webserver/ui). + +--- + +## Usage + +Ready to get started with Dagster Pipes? Check out the Dagster Pipes tutorial to get up and running! diff --git a/docs/content/guides/dagster-pipes/modifying-existing-code-to-work-with-dagster-pipes.mdx b/docs/content/guides/dagster-pipes/modifying-existing-code-to-work-with-dagster-pipes.mdx deleted file mode 100644 index fcb8570e5745c..0000000000000 --- a/docs/content/guides/dagster-pipes/modifying-existing-code-to-work-with-dagster-pipes.mdx +++ /dev/null @@ -1,6 +0,0 @@ ---- -title: "Modifying existing code to work with Dagster Pipes | Dagster Docs" -description: "" ---- - -# Modifying existing code to work with Dagster Pipes diff --git a/docs/content/integrations/embedded-elt.mdx b/docs/content/integrations/embedded-elt.mdx index 74ad4cef0181d..1d60e0d193ab5 100644 --- a/docs/content/integrations/embedded-elt.mdx +++ b/docs/content/integrations/embedded-elt.mdx @@ -109,6 +109,9 @@ sling_job = build_assets_job( This is an example of how to setup a Sling sync between Postgres and Snowflake: ```python file=/integrations/embedded_elt/postgres_snowflake.py +# pyright: reportGeneralTypeIssues=none +# pyright: reportOptionalMemberAccess=none + import os from dagster_embedded_elt.sling import ( diff --git a/docs/next/public/images/guides/dagster-pipes/dagster-pipes-process.png b/docs/next/public/images/guides/dagster-pipes/dagster-pipes-process.png new file mode 100644 index 0000000000000..e9a0ce37c95c6 Binary files /dev/null and b/docs/next/public/images/guides/dagster-pipes/dagster-pipes-process.png differ