concepts/dagster-pipes

dagster-io · Mar 25, 2024 · c112cc4 · c112cc4
1 parent b54599d
commit c112cc4
Show file tree

Hide file tree

Showing 6 changed files with 51 additions and 51 deletions.
diff --git a/docs/content/guides/dagster-pipes.mdx → docs/content/concepts/dagster-pipes.mdx b/docs/content/guides/dagster-pipes.mdx → docs/content/concepts/dagster-pipes.mdx
diff --git a/docs/content/concepts/dagster-pipes/aws-lambda.mdx b/docs/content/concepts/dagster-pipes/aws-lambda.mdx
@@ -64,10 +64,10 @@ For simplicity we're demonstrating this step using the UI, but you can also do t
 After the function is created, you'll be redirected to a page like the following:
 
 <Image
-  alt="The Lambda function details page in the AWS UI"
-  src="/images/guides/dagster-pipes/aws-lambda/aws-lambda-function-details.png"
-  width={1920}
-  height={1153}
+alt="The Lambda function details page in the AWS UI"
+src="/images/guides/dagster-pipes/aws-lambda/aws-lambda-function-details.png"
+width={1920}
+height={1153}
 />
 
 ### Step 1.2: Add a dagster-pipes file
@@ -86,10 +86,10 @@ Next, you'll add `dagster-pipes` to the function.
 1. In the **Code source** section of the page, add a new file. This can be accomplished with **File > New file** or by clicking the green **+** icon next to the open `lambda_function` tab:
 
    <Image
-     alt="Highlighted New file option in a menu of the Code source section on the Lambda function details page"
-     src="/images/guides/dagster-pipes/aws-lambda/aws-lambda-create-new-file.png"
-     width={2386}
-     height={1196}
+   alt="Highlighted New file option in a menu of the Code source section on the Lambda function details page"
+   src="/images/guides/dagster-pipes/aws-lambda/aws-lambda-create-new-file.png"
+   width={2386}
+   height={1196}
    />
 
 2. **In a new browser tab**, navigate to the following URL:
@@ -101,10 +101,10 @@ Next, you'll add `dagster-pipes` to the function.
 3. Copy the contents of `__init__.py` into the new file you created in AWS. **Note**: We recommend adding the source link and the date you copied the contents to the top of the file as comments:
 
    <Image
-     alt="The copied contents of the Dagster Pipes file into a file in the AWS UI"
-     src="/images/guides/dagster-pipes/aws-lambda/aws-lambda-add-dagster-pipes.png"
-     width={2390}
-     height={1090}
+   alt="The copied contents of the Dagster Pipes file into a file in the AWS UI"
+   src="/images/guides/dagster-pipes/aws-lambda/aws-lambda-add-dagster-pipes.png"
+   width={2390}
+   height={1090}
    />
 
 4. Save the file as `dagster_pipes.py`.
@@ -244,10 +244,10 @@ In this step, you’ll invoke the AWS Lambda function you defined in [Step 1](#s
    <!-- ![Event log for AWS Lambda run](/images/guides/dagster-pipes/aws-lambda/aws-lambda-dagster-ui.png) -->
 
    <Image
-     alt="Event log for AWS Lambda run"
-     src="/images/guides/dagster-pipes/aws-lambda/aws-lambda-dagster-ui.png"
-     width={1501}
-     height={1416}
+   alt="Event log for AWS Lambda run"
+   src="/images/guides/dagster-pipes/aws-lambda/aws-lambda-dagster-ui.png"
+   width={1501}
+   height={1416}
    />
 
 ---

diff --git a/docs/content/concepts/dagster-pipes/dagster-pipes-details-and-customization.mdx b/docs/content/concepts/dagster-pipes/dagster-pipes-details-and-customization.mdx
@@ -16,10 +16,10 @@ This guide will cover the lower level Pipes APIs and how you can compose them to
 <!-- ![Detailed overview of a Dagster Pipes session](/images/guides/dagster-pipes/pipes-overview.png) -->
 
 <Image
-  alt="Detailed overview of a Dagster Pipes session"
-  src="/images/guides/dagster-pipes/pipes-overview.png"
-  width={1300}
-  height={510}
+alt="Detailed overview of a Dagster Pipes session"
+src="/images/guides/dagster-pipes/pipes-overview.png"
+width={1300}
+height={510}
 />
 
 <table
@@ -177,7 +177,7 @@ A **Pipes session** is the time spanning:
 
 There are separate APIs for interacting with a Pipes session in the orchestration and external processes. The orchestration process API is defined in `dagster`. The external process API is defined by a Pipes integration library that is loaded by user code in the external process. This library knows how to interpret the bootstrap payload and spin up a context loader and message writer.
 
-At present the only official Dagster Pipes integration library is Python’s [`dagster-pipes`](/_apidocs/libraries/dagster-pipes), available on [PyPI](https://pypi.org/project/dagster-pipes/). The library has no dependencies and fits in a [single file](https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/__init__.py), so it may also be trivially vendored.
+At present the only official Dagster Pipes integration library is Python’s [`dagster-pipes`](/\_apidocs/libraries/dagster-pipes), available on [PyPI](https://pypi.org/project/dagster-pipes/). The library has no dependencies and fits in a [single file](https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-pipes/dagster_pipes/\__init\_\_.py), so it may also be trivially vendored.
 
 ### Session lifecycle (orchestration process)
 
@@ -251,7 +251,7 @@ For <PyObject object="open_pipes_session" />, these three parts perform the foll
 
 ### Session lifecycle (external process)
 
-As noted above, currently the only existing Pipes integration library is Python’s [`dagster-pipes`](/_apidocs/libraries/dagster-pipes). The below example therefore uses Python and `dagster-pipes`. In the future we will be releasing `dagster-pipes` equivalents for selected other languages. and the concepts illustrated here should map straightforwardly to these other integration libraries.
+As noted above, currently the only existing Pipes integration library is Python’s [`dagster-pipes`](/\_apidocs/libraries/dagster-pipes). The below example therefore uses Python and `dagster-pipes`. In the future we will be releasing `dagster-pipes` equivalents for selected other languages. and the concepts illustrated here should map straightforwardly to these other integration libraries.
 
 A Pipes session is represented in the external process by a <PyObject object="PipesContext" module="dagster_pipes" /> object. A session created by the launching orchestration process can be connected to with <PyObject object="open_dagster_pipes" module="dagster_pipes" /> from `dagster-pipes`:
 

diff --git a/docs/content/concepts/dagster-pipes/databricks.mdx b/docs/content/concepts/dagster-pipes/databricks.mdx
@@ -132,7 +132,7 @@ Let's review what's happening in this code:
 
 ### Step 1.2: Define the Databricks Pipes client and definitions
 
-The [`dagster-databricks`](/_apidocs/libraries/dagster-databricks) library provides a <PyObject module="dagster_databricks" object="PipesDatabricksClient" />, which is a pre-built Dagster resource that allows you to quickly get Pipes working with your Databricks workspace.
+The [`dagster-databricks`](/\_apidocs/libraries/dagster-databricks) library provides a <PyObject module="dagster_databricks" object="PipesDatabricksClient" />, which is a pre-built Dagster resource that allows you to quickly get Pipes working with your Databricks workspace.
 
 Add the following to the bottom of `dagster_databricks_pipes.py` to define the resource and a <PyObject object="Definitions" /> object that binds it to the `databricks_asset`:
 
@@ -263,19 +263,19 @@ In this step, you’ll run the Databricks job you created in [Step 1.2](#step-12
 2. Navigate to [localhost:3000](http://localhost:3000/), where you should see the UI:
 
    <Image
-     alt="Databricks asset"
-     src="/images/guides/dagster-pipes/databricks/asset.png"
-     width={2554}
-     height={1335}
+   alt="Databricks asset"
+   src="/images/guides/dagster-pipes/databricks/asset.png"
+   width={2554}
+   height={1335}
    />
 
 3. Click **Materialize** near the top right corner of the page, then click **View** on the **Launched Run** popup. Wait for the run to complete, and the event log should look like this:
 
    <Image
-     alt="Event log for Databricks run"
-     src="/images/guides/dagster-pipes/databricks/run.png"
-     width={2554}
-     height={1335}
+   alt="Event log for Databricks run"
+   src="/images/guides/dagster-pipes/databricks/run.png"
+   width={2554}
+   height={1335}
    />
 
 ---

diff --git a/docs/content/concepts/dagster-pipes/kubernetes.mdx b/docs/content/concepts/dagster-pipes/kubernetes.mdx
@@ -224,10 +224,10 @@ In this step, you’ll run the Kubernetes container you defined in [Step 1](#ste
 3. Click **Materialize** near the top right corner of the page, then click **View** on the **Launched Run** popup. Wait for the run to complete, and the event log should look like this:
 
    <Image
-     alt="Event log for Kubernetes run"
-     src="/images/guides/dagster-pipes/kubernetes/run.png"
-     width={1784}
-     height={1495}
+   alt="Event log for Kubernetes run"
+   src="/images/guides/dagster-pipes/kubernetes/run.png"
+   width={1784}
+   height={1495}
    />
 
 ---

diff --git a/docs/content/concepts/dagster-pipes/subprocess/create-subprocess-asset.mdx b/docs/content/concepts/dagster-pipes/subprocess/create-subprocess-asset.mdx
@@ -163,37 +163,37 @@ In this step, you’ll execute the subprocess asset you created in earlier steps
 2. Navigate to <http://localhost:3000>, where you should see the UI:
 
    <Image
-     alt="Asset in the UI"
-     src="/images/guides/dagster-pipes/subprocess/part-1-step-3-2-asset.png"
-     width={1264}
-     height={714}
+   alt="Asset in the UI"
+   src="/images/guides/dagster-pipes/subprocess/part-1-step-3-2-asset.png"
+   width={1264}
+   height={714}
    />
 
 3. Click **Materialize** located in the top right to run your code:
 
    <Image
-     alt="Materialize asset"
-     src="/images/guides/dagster-pipes/subprocess/part-1-step-3-3-materialize.png"
-     width={1326}
-     height={744}
+   alt="Materialize asset"
+   src="/images/guides/dagster-pipes/subprocess/part-1-step-3-3-materialize.png"
+   width={1326}
+   height={744}
    />
 
 4. Navigate to the [Run details](/concepts/webserver/ui#run-details) page, where you should see the logs for the run:
 
    <Image
-     alt="Logs in the run details page"
-     src="/images/guides/dagster-pipes/subprocess/part-1-step-3-4-logs.png"
-     width={1272}
-     height={718}
+   alt="Logs in the run details page"
+   src="/images/guides/dagster-pipes/subprocess/part-1-step-3-4-logs.png"
+   width={1272}
+   height={718}
    />
 
 5. In `external_code.py`, we have a `print` statement that outputs to `stdout`. Dagster will display these in the UI's raw compute log view. To see the `stdout` log, toggle the log section to **stdout**:
 
    <Image
-     alt="Raw compute logs in the run details page"
-     src="/images/guides/dagster-pipes/subprocess/part-1-step-3-5-stdout.png"
-     width={1272}
-     height={718}
+   alt="Raw compute logs in the run details page"
+   src="/images/guides/dagster-pipes/subprocess/part-1-step-3-5-stdout.png"
+   width={1272}
+   height={718}
    />
 
 ---