docs nav RFC: move pipes to concepts (dagster-io#20712)

## Summary & Motivation this PR moves pipes pages to Concepts - updates all internal links - updates nav -- open to suggestions to the exact order of the nav - sets up redirects so /guides/dagster-pipes URLs out there would still work ## How I Tested These Changes preview: https://03-25-docs-move-pipes-to-concepts.dagster.dagster-docs.io/concepts/dagster-pipes <img width="327" alt="image" src="https://github.com/dagster-io/dagster/assets/4531914/0090db02-0d2c-43ac-8f91-2728c91430e2">
danielgafni · Jun 18, 2024 · 9df8c5f · 9df8c5f
1 parent 9c9b2fe
commit 9df8c5f
Show file tree

Hide file tree

Showing 15 changed files with 120 additions and 94 deletions.
diff --git a/docs/content/_apidocs.mdx b/docs/content/_apidocs.mdx
@@ -206,7 +206,7 @@ APIs from the core `dagster` package, divided roughly by topic:
       </td>
       <td>
         APIs for working with the{" "}
-        <a href="/guides/dagster-pipes">Dagster Pipes protocol</a> from the
+        <a href="/concepts/dagster-pipes">Dagster Pipes protocol</a> from the
         orchestration side.
       </td>
     </tr>

diff --git a/docs/content/_navigation.json b/docs/content/_navigation.json
@@ -305,6 +305,46 @@
               }
             ]
           },
+          {
+            "title": "Dagster Pipes (Experimental)",
+            "path": "/concepts/dagster-pipes",
+            "children": [
+              {
+                "title": "Dagster Pipes tutorial",
+                "path": "/concepts/dagster-pipes/subprocess",
+                "children": [
+                  {
+                    "title": "Part 1: Define a Dagster asset",
+                    "path": "/concepts/dagster-pipes/subprocess/create-subprocess-asset"
+                  },
+                  {
+                    "title": "Part 2: Modify external code",
+                    "path": "/concepts/dagster-pipes/subprocess/modify-external-code"
+                  },
+                  {
+                    "title": "Dagster Pipes + subprocess reference",
+                    "path": "/concepts/dagster-pipes/subprocess/reference"
+                  }
+                ]
+              },
+              {
+                "title": "Dagster Pipes + AWS Lambda",
+                "path": "/concepts/dagster-pipes/aws-lambda"
+              },
+              {
+                "title": "Dagster Pipes + Databricks",
+                "path": "/concepts/dagster-pipes/databricks"
+              },
+              {
+                "title": "Dagster Pipes + Kubernetes",
+                "path": "/concepts/dagster-pipes/kubernetes"
+              },
+              {
+                "title": "Details and customization",
+                "path": "/concepts/dagster-pipes/dagster-pipes-details-and-customization"
+              }
+            ]
+          },
           {
             "title": "I/O management",
             "children": [
@@ -685,7 +725,7 @@
                 "path": "/dagster-plus/managing-deployments/branch-deployments",
                 "children": [
                   {
-                   "title": "Overview",
+                    "title": "Overview",
                     "path": "/dagster-plus/managing-deployments/branch-deployments"
                   },
                   {
@@ -701,7 +741,7 @@
                     "path": "/dagster-plus/managing-deployments/branch-deployments/using-branch-deployments"
                   },
                   {
-                   "title": "Change Tracking",
+                    "title": "Change Tracking",
                     "path": "/dagster-plus/managing-deployments/branch-deployments/change-tracking"
                   }
                 ]
@@ -1057,46 +1097,6 @@
           }
         ]
       },
-      {
-        "title": "Dagster Pipes (Experimental)",
-        "path": "/guides/dagster-pipes",
-        "children": [
-          {
-            "title": "Dagster Pipes tutorial",
-            "path": "/guides/dagster-pipes/subprocess",
-            "children": [
-              {
-                "title": "Part 1: Define a Dagster asset",
-                "path": "/guides/dagster-pipes/subprocess/create-subprocess-asset"
-              },
-              {
-                "title": "Part 2: Modify external code",
-                "path": "/guides/dagster-pipes/subprocess/modify-external-code"
-              },
-              {
-                "title": "Dagster Pipes & subprocess reference",
-                "path": "/guides/dagster-pipes/subprocess/reference"
-              }
-            ]
-          },
-          {
-            "title": "Dagster Pipes & AWS Lambda",
-            "path": "/guides/dagster-pipes/aws-lambda"
-          },
-          {
-            "title": "Dagster Pipes & Databricks",
-            "path": "/guides/dagster-pipes/databricks"
-          },
-          {
-            "title": "Dagster Pipes & Kubernetes",
-            "path": "/guides/dagster-pipes/kubernetes"
-          },
-          {
-            "title": "Details and customization",
-            "path": "/guides/dagster-pipes/dagster-pipes-details-and-customization"
-          }
-        ]
-      },
       {
         "title": "Version migration",
         "path": "/migration",

diff --git a/docs/content/concepts.mdx b/docs/content/concepts.mdx
@@ -211,6 +211,37 @@ Apply tags and metadata to organize your project and provide useful context to o
   ></ArticleListItem>
 </ArticleList>
 
+### Dagster Pipes (Experimental)
+
+Dagster Pipes is a toolkit for building integrations between Dagster and external execution environments.
+
+<ArticleList>
+  <ArticleListItem
+    title="Dagster Pipes"
+    href="/concepts/dagster-pipes"
+  ></ArticleListItem>
+  <ArticleListItem
+    title="Dagster Pipes tutorial"
+    href="/concepts/dagster-pipes/subprocess"
+  ></ArticleListItem>
+  <ArticleListItem
+    title="Dagster Pipes + AWS Lambda"
+    href="/concepts/dagster-pipes/aws-lambda"
+  ></ArticleListItem>
+  <ArticleListItem
+    title="Dagster Pipes + Databricks"
+    href="/concepts/dagster-pipes/databricks"
+  ></ArticleListItem>
+  <ArticleListItem
+    title="Dagster Pipes + Kubernetes"
+    href="/concepts/dagster-pipes/kubernetes"
+  ></ArticleListItem>
+  <ArticleListItem
+    title="Dagster Pipes details and customization"
+    href="/concepts/dagster-pipes/dagster-pipes-details-and-customization"
+  ></ArticleListItem>
+</ArticleList>
+
 ### I/O management
 
 I/O managers are user-provided objects that store asset and op outputs and load them as inputs to downstream assets and ops.

diff --git a/docs/content/guides/dagster-pipes.mdx → docs/content/concepts/dagster-pipes.mdx b/docs/content/guides/dagster-pipes.mdx → docs/content/concepts/dagster-pipes.mdx
@@ -74,12 +74,12 @@ While Dagster Pipes is lightweight and flexible, there are a few limitations to
 
 Ready to get started with Dagster Pipes? Depending on what your goal is, how you approach using Dagster Pipes can differ.
 
-- **If you’re writing scripts that are being orchestrated**, all you’ll need to do is lightly modify your existing scripts. Check out the [Dagster Pipes tutorial](/guides/dagster-pipes/subprocess) to get up and running.
+- **If you’re writing scripts that are being orchestrated**, all you’ll need to do is lightly modify your existing scripts. Check out the [Dagster Pipes tutorial](/concepts/dagster-pipes/subprocess) to get up and running.
 
 - **If you need to orchestrate scripts others have written**, you can use our ready-to-go integrations to execute scripts in your chosen technology:
 
-  - [Amazon Web Services Lambda](/guides/dagster-pipes/aws-lambda)
-  - [Databricks](/guides/dagster-pipes/databricks)
-  - [Kubernetes](/guides/dagster-pipes/kubernetes)
+  - [Amazon Web Services Lambda](/concepts/dagster-pipes/aws-lambda)
+  - [Databricks](/concepts/dagster-pipes/databricks)
+  - [Kubernetes](/concepts/dagster-pipes/kubernetes)
 
-- **If you don’t see your integration or you want to fully customize your Pipes experience**, check out the [Dagster Pipes details and customization guide](/guides/dagster-pipes/dagster-pipes-details-and-customization) to learn how to create a custom experience.
+- **If you don’t see your integration or you want to fully customize your Pipes experience**, check out the [Dagster Pipes details and customization guide](/concepts/dagster-pipes/dagster-pipes-details-and-customization) to learn how to create a custom experience.
diff --git a/...ntent/guides/dagster-pipes/aws-lambda.mdx → ...ent/concepts/dagster-pipes/aws-lambda.mdx b/...ntent/guides/dagster-pipes/aws-lambda.mdx → ...ent/concepts/dagster-pipes/aws-lambda.mdx
@@ -8,13 +8,13 @@ description: "Learn to integrate Dagster Pipes with AWS Lambda to launch externa
 <Note>
   <strong>Heads up!</strong> This guide focuses on using an out-of-the-box
   Amazon Web Services (AWS) Lambda resource. For further customization, use the{" "}
-  <a href="/guides/dagster-pipes/dagster-pipes-details-and-customization">
+  <a href="/concepts/dagster-pipes/dagster-pipes-details-and-customization">
     <code>open_pipes_session</code> approach
   </a>{" "}
   instead.
 </Note>
 
-In this guide, we’ll show you how to use [Dagster Pipes](/guides/dagster-pipes) with Dagster’s AWS Lambda integration to invoke a Lambda function and execute external code.
+In this guide, we’ll show you how to use [Dagster Pipes](/concepts/dagster-pipes) with Dagster’s AWS Lambda integration to invoke a Lambda function and execute external code.
 
 Pipes allows your code to interact with Dagster outside of a full Dagster environment. Instead, the environment only needs to contain `dagster-pipes`, a single-file Python package with no dependencies that can be installed from PyPI or easily vendored. `dagster-pipes` handles streaming `stdout`/`stderr` and Dagster events back to the orchestration process.
 
@@ -154,7 +154,7 @@ Let's review what this code does:
 
   We're using the default context loader (<PyObject object="PipesDefaultContextLoader" module="dagster_pipes" />) and message writer (<PyObject object="PipesDefaultMessageWriter" module="dagster_pipes" />) in this example. These objects establish communication between the orchestration and external process. On the orchestration end, these match a corresponding `PipesLambdaEventContextInjector` and `PipesLambdaLogsMessageReader`, which are instantiated inside the <PyObject module="dagster_aws.pipes" object="PipesLambdaClient" />.
 
-- **Inside the body of the context manager (<PyObject object="open_dagster_pipes" module="dagster_pipes" />), retrieve a log and report an asset materialization.** These calls use the temporary communications channels established by <PyObject object="PipesDefaultContextLoader" module="dagster_pipes" /> and <PyObject object="PipesDefaultMessageWriter" module="dagster_pipes" />. To see the full range of what you can do with the <PyObject object="PipesContext" module="dagster_pipes" />, see the API docs or the general [Pipes documentation](/guides/dagster-pipes).
+- **Inside the body of the context manager (<PyObject object="open_dagster_pipes" module="dagster_pipes" />), retrieve a log and report an asset materialization.** These calls use the temporary communications channels established by <PyObject object="PipesDefaultContextLoader" module="dagster_pipes" /> and <PyObject object="PipesDefaultMessageWriter" module="dagster_pipes" />. To see the full range of what you can do with the <PyObject object="PipesContext" module="dagster_pipes" />, see the API docs or the general [Pipes documentation](/concepts/dagster-pipes).
 
 At this point you can execute the rest of your AWS Lambda code as normal, invoking various <PyObject object="PipesContext" module="dagster_pipes" /> APIs as needed.
 
@@ -257,11 +257,11 @@ In this step, you’ll invoke the AWS Lambda function you defined in [Step 1](#s
 <ArticleList>
   <ArticleListItem
     title="Dagster Pipes"
-    href="/guides/dagster-pipes"
+    href="/concepts/dagster-pipes"
   ></ArticleListItem>
   <ArticleListItem
     title="Dagster Pipes details and customization"
-    href="/guides/dagster-pipes/dagster-pipes-details-and-customization"
+    href="/concepts/dagster-pipes/dagster-pipes-details-and-customization"
   ></ArticleListItem>
   <ArticleListItem
     title="dagster-aws API reference"

diff --git a/...gster-pipes-details-and-customization.mdx → ...gster-pipes-details-and-customization.mdx b/...gster-pipes-details-and-customization.mdx → ...gster-pipes-details-and-customization.mdx
@@ -5,7 +5,7 @@ description: "Learn about Dagster Pipes APIs and how to compose them to create a
 
 # Dagster Pipes details and customization
 
-[Dagster Pipes](/guides/dagster-pipes) is a toolkit for integrating Dagster with an arbitrary external compute environment. While many users will be well-served by the simplified interface offered by Pipes client objects (e.g. <PyObject object="PipesSubprocessClient" />, <PyObject object="PipesDatabricksClient" module="dagster_databricks"/>), others will need a greater level of control over Pipes. This is particularly the case for users seeking to connect large existing codebases to Dagster.
+[Dagster Pipes](/concepts/dagster-pipes) is a toolkit for integrating Dagster with an arbitrary external compute environment. While many users will be well-served by the simplified interface offered by Pipes client objects (e.g. <PyObject object="PipesSubprocessClient" />, <PyObject object="PipesDatabricksClient" module="dagster_databricks"/>), others will need a greater level of control over Pipes. This is particularly the case for users seeking to connect large existing codebases to Dagster.
 
 This guide will cover the lower level Pipes APIs and how you can compose them to provide a custom solution for your data platform.
 
@@ -487,10 +487,10 @@ class MyCustomCloudServiceMessageWriterChannel(PipesBlobStoreMessageWriterChanne
 <ArticleList>
   <ArticleListItem
     title="Dagster Pipes"
-    href="/guides/dagster-pipes"
+    href="/concepts/dagster-pipes"
   ></ArticleListItem>
   <ArticleListItem
     title="Dagster Pipes tutorial"
-    href="/guides/dagster-pipes/subprocess"
+    href="/concepts/dagster-pipes/subprocess"
   ></ArticleListItem>
 </ArticleList>
diff --git a/...ntent/guides/dagster-pipes/databricks.mdx → ...ent/concepts/dagster-pipes/databricks.mdx b/...ntent/guides/dagster-pipes/databricks.mdx → ...ent/concepts/dagster-pipes/databricks.mdx
@@ -5,7 +5,7 @@ description: "Learn to integrate Dagster Pipes with Databricks to launch externa
 
 # Integrating Databricks with Dagster Pipes
 
-In this guide, we’ll show you how to use [Dagster Pipes](/guides/dagster-pipes) with Dagster’s Databricks integration to launch Databricks jobs.
+In this guide, we’ll show you how to use [Dagster Pipes](/concepts/dagster-pipes) with Dagster’s Databricks integration to launch Databricks jobs.
 
 Pipes allows your Databricks jobs to stream logs (including `stdout` and `stderr` of the driver process) and events back to Dagster. This does not require a full Dagster environment on Databricks; instead:
 
@@ -210,13 +210,13 @@ with open_dagster_pipes(
 
 Before we go any futher, let's review what this script does:
 
-- **Imports `PipesDbfsContextLoader`, `PipesDbfsMessageWriter`, and `open_dagster_pipes` from `dagster_pipes`.** The <PyObject object="PipesDbfsContextLoader" module="dagster_pipes" /> and <PyObject object="PipesDbfsMessageWriter" module="dagster_pipes" /> are DBFS-specific implementations of the <PyObject object="PipesContextLoader" /> and <PyObject object="PipesMessageWriter" />. Refer to the [Dagster Pipes details and customization Guide](/guides/dagster-pipes/dagster-pipes-details-and-customization) for protocol details.
+- **Imports `PipesDbfsContextLoader`, `PipesDbfsMessageWriter`, and `open_dagster_pipes` from `dagster_pipes`.** The <PyObject object="PipesDbfsContextLoader" module="dagster_pipes" /> and <PyObject object="PipesDbfsMessageWriter" module="dagster_pipes" /> are DBFS-specific implementations of the <PyObject object="PipesContextLoader" /> and <PyObject object="PipesMessageWriter" />. Refer to the [Dagster Pipes details and customization Guide](/concepts/dagster-pipes/dagster-pipes-details-and-customization) for protocol details.
 
   Both objects write temporary files on DBFS for communication between the orchestration and external process. The <PyObject object="PipesDbfsContextLoader" module="dagster_pipes" /> and <PyObject object="PipesDbfsMessageWriter" module="dagster_pipes" /> match a corresponding `PipesDbfsContextInjector` and `PipesDbfsMessageReader` on the orchestration end, which are instantiated inside the <PyObject object="PipesDatabricksClient" module="dagster_databricks" />.
 
 - **Passes the context loader and message writer to the <PyObject object="open_dagster_pipes" module="dagster_pipes" /> context manager**, which yields an instance of <PyObject object="PipesContext" module="dagster_pipes" /> called `pipes`.
 
-  Inside the body of the context manager are various calls against `pipes` to retrieve an extra, log, and report an asset materialization. All of these calls will use the DBFS temporary file-based communications channels established by <PyObject object="PipesDbfsContextLoader" module="dagster_pipes" /> and <PyObject object="PipesDbfsMessageWriter" module="dagster_pipes" />. To see the full range of what you can do with the <PyObject object="PipesContext" module="dagster_pipes" />, see the API docs or the general [Pipes guide](/guides/dagster-pipes).
+  Inside the body of the context manager are various calls against `pipes` to retrieve an extra, log, and report an asset materialization. All of these calls will use the DBFS temporary file-based communications channels established by <PyObject object="PipesDbfsContextLoader" module="dagster_pipes" /> and <PyObject object="PipesDbfsMessageWriter" module="dagster_pipes" />. To see the full range of what you can do with the <PyObject object="PipesContext" module="dagster_pipes" />, see the API docs or the general [Pipes guide](/concepts/dagster-pipes).
 
 At this point you can execute the rest of your Databricks code as normal, invoking various <PyObject object="PipesContext" module="dagster_pipes" /> APIs as needed.
 
@@ -378,11 +378,11 @@ def databricks_asset(context: AssetExecutionContext):
 <ArticleList>
   <ArticleListItem
     title="Dagster Pipes"
-    href="/guides/dagster-pipes"
+    href="/concepts/dagster-pipes"
   ></ArticleListItem>
   <ArticleListItem
     title="Dagster Pipes details and customization"
-    href="/guides/dagster-pipes/dagster-pipes-details-and-customization"
+    href="/concepts/dagster-pipes/dagster-pipes-details-and-customization"
   ></ArticleListItem>
   <ArticleListItem
     title="dagster-databricks API reference"

diff --git a/...integrating-docker-with-dagster-pipes.mdx → ...integrating-docker-with-dagster-pipes.mdx b/...integrating-docker-with-dagster-pipes.mdx → ...integrating-docker-with-dagster-pipes.mdx