[Docs] Add structure for remaining missing content (Dagster+) (dagste…

…r-io#26299) ## Summary & Motivation * Adds (mostly empty) directories and files for remaining Dagster+ content and adds TODOs and links to old content to be copied over * Makes metadata consistent across files so future docs contributors don't needlessly copy metadata that doesn't matter into new docs * Reorganizes deployment docs into "Deployment types", "Management", and "Migration" categories * Moves "Code locations" section to "Features" * Creates "CI/CD" section in "Features" and moves "Branch deployments" into it ## How I Tested These Changes Local and staging deployment ## Changelog --------- Signed-off-by: nikki everett <[email protected]>
pskinnerthyme · Dec 16, 2024 · 635155e · 635155e
1 parent 13b4bbc
commit 635155e
Show file tree

Hide file tree

Showing 106 changed files with 723 additions and 706 deletions.
diff --git a/.../docs-beta/docs/dagster-plus/deployment/code-locations/code-location-history.md b/.../docs-beta/docs/dagster-plus/deployment/code-locations/code-location-history.md
@@ -1,12 +1,8 @@
 ---
 title: "Code location history and rollbacks"
-displayed_sidebar: "dagsterPlus"
 sidebar_position: 100
-sidebar_label: "Code location history and rollbacks"
 ---
 
-# Code location history and rollbacks
-
 Dagster+ automatically tracks metadata every time a code location is loaded. This can be used to understand when changes have been made, and what those changes were. In addition, this metadata can be used to quickly redeploy an older version.
 
 <details>
@@ -54,4 +50,4 @@ If you notice an issue with newly deployed code, or your code fails to deploy su
 ## Next steps
 
 - Learn more about [Code Locations](/dagster-plus/deployment/code-locations)
-- Learn how to [Alert when a code location fails to load](/dagster-plus/features/alerts#alerting-when-a-code-location-fails-to-load)
+- Learn how to [Alert when a code location fails to load](/dagster-plus/features/alerts/creating-alerts#alerting-when-a-code-location-fails-to-load)
diff --git a/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md b/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md
@@ -0,0 +1,7 @@
+---
+title: dagster_cloud.yaml reference
+sidebar_position: 200
+unlisted: true
+---
+
+{/* TODO move content from https://docs.dagster.io/dagster-plus/managing-deployments/dagster-cloud-yaml */}
diff --git a/docs/docs-beta/docs/dagster-plus/deployment/code-locations/index.md b/docs/docs-beta/docs/dagster-plus/deployment/code-locations/index.md
@@ -1,11 +1,8 @@
 ---
 title: "Code locations"
-displayed_sidebar: "dagsterPlus"
-sidebar_position: 50
+sidebar_position: 40
 ---
 
-# Code locations
-
 Separate code locations allow you to deploy different projects that still roll up into a single Dagster+ deployment with one global lineage graph.
 
 This guide will cover three options for adding a new code location:
@@ -237,4 +234,4 @@ The monorepo should have CI/CD configured to deploy your changes and add or upda
 ## Next steps
 
 - After adding a code location, you may want to setup access controls
-- You may want to add additional configuration to your code location. This configuration will vary by agent type, but see examples for [setting default resource limits for Kubernetes](/dagster-plus/deployment/hybrid/agents/kubernetes) or [changing the IAM role for ECS](/todo).
+- You may want to add additional configuration to your code location. This configuration will vary by agent type, but see examples for [setting default resource limits for Kubernetes](/dagster-plus/deployment/deployment-types/hybrid/kubernetes) or [changing the IAM role for ECS](/todo).
diff --git a/docs/docs-beta/docs/dagster-plus/deployment/code-requirements.md b/docs/docs-beta/docs/dagster-plus/deployment/code-requirements.md
@@ -1,8 +1,7 @@
 ---
 title: 'Dagster+ code requirements'
-displayed_sidebar: 'dagsterPlus'
 sidebar_label: "Code requirements"
-sidebar_position: 100
+sidebar_position: 10
 ---
 
 Your Dagster project must meet a few requirements to run in Dagster+.
@@ -34,7 +33,7 @@ To work with Dagster+, your Dagster code:
 
 ## Hybrid deployment requirements
 
-If you're using [Hybrid Deployment](/dagster-plus/deployment/hybrid), there are a few additional requirements.
+If you're using [Hybrid Deployment](/dagster-plus/deployment/deployment-types/hybrid), there are a few additional requirements.
 
 - **If using an Amazon Elastic Container Service (ECS), Kubernetes, or Docker agent**, your code must be packaged into a Docker image and pushed to a registry your agent can access. Dagster+ doesn't need access to your image - the agent only needs to be able to pull it.
 

diff --git a/docs/docs-beta/docs/dagster-plus/deployment/deployment-settings.md b/docs/docs-beta/docs/dagster-plus/deployment/deployment-settings.md
diff --git a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types.md b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types.md
diff --git a/...r-plus/deployment/deployment-types/hybrid/amazon-ecs/configuration-reference.md b/...r-plus/deployment/deployment-types/hybrid/amazon-ecs/configuration-reference.md
@@ -0,0 +1,7 @@
+---
+title: Configuration reference
+sidebar_position: 400
+unlisted: true
+---
+
+{/* TODO copy from https://docs.dagster.io/dagster-plus/deployment/agents/amazon-ecs/configuration-reference */}
diff --git a/...docs/dagster-plus/deployment/deployment-types/hybrid/amazon-ecs/existing-vpc.md b/...docs/dagster-plus/deployment/deployment-types/hybrid/amazon-ecs/existing-vpc.md
@@ -0,0 +1,7 @@
+---
+title: Existing VPC setup
+sidebar_position: 200
+unlisted: true
+---
+
+{/* TODO copy from https://docs.dagster.io/dagster-plus/deployment/agents/amazon-ecs/creating-ecs-agent-existing-vpc */}
diff --git a/...s-beta/docs/dagster-plus/deployment/deployment-types/hybrid/amazon-ecs/index.md b/...s-beta/docs/dagster-plus/deployment/deployment-types/hybrid/amazon-ecs/index.md
@@ -0,0 +1,8 @@
+---
+title: Amazon ECS agent
+sidebar_position: 50
+---
+
+import DocCardList from '@theme/DocCardList';
+
+<DocCardList />
diff --git a/.../dagster-plus/deployment/deployment-types/hybrid/amazon-ecs/manual-provision.md b/.../dagster-plus/deployment/deployment-types/hybrid/amazon-ecs/manual-provision.md
@@ -0,0 +1,7 @@
+---
+title: Manual provision setup
+sidebar_position: 300
+unlisted: true
+---
+
+{/* TODO move content from https://docs.dagster.io/dagster-plus/deployment/agents/amazon-ecs/manually-provisioning-ecs-agent */}
diff --git a/...beta/docs/dagster-plus/deployment/deployment-types/hybrid/amazon-ecs/new-vpc.md b/...beta/docs/dagster-plus/deployment/deployment-types/hybrid/amazon-ecs/new-vpc.md
@@ -0,0 +1,7 @@
+---
+title: New VPC setup
+sidebar_position: 100
+unlisted: true
+---
+
+{/* TODO copy from https://docs.dagster.io/dagster-plus/deployment/agents/amazon-ecs/creating-ecs-agent-new-vpc */}
diff --git a/...-plus/deployment/deployment-types/hybrid/amazon-ecs/upgrading-cloudformation.md b/...-plus/deployment/deployment-types/hybrid/amazon-ecs/upgrading-cloudformation.md
@@ -0,0 +1,8 @@
+---
+title: Upgrading CloudFormation for an Amazon ECS agent
+sidebar_label: Upgrading CloudFormation
+sidebar_position: 500
+unlisted: true
+---
+
+{/* TODO move from https://docs.dagster.io/dagster-plus/deployment/agents/amazon-ecs/upgrading-cloudformation-template */}
diff --git a/...er-plus/deployment/hybrid/architecture.md → ...t/deployment-types/hybrid/architecture.md b/...er-plus/deployment/hybrid/architecture.md → ...t/deployment-types/hybrid/architecture.md
@@ -1,11 +1,8 @@
 ---
 title: 'Dagster+ Hybrid architecture'
-displayed_sidebar: 'dagsterPlus'
-sidebar_position: 100
+sidebar_position: 10
 ---
 
-# Dagster+ Hybrid architecture
-
 The Hybrid architecture is the most flexible and secure way to deploy Dagster+. It allows you to run your user code in your environment while leveraging Dagster+'s infrastructure for orchestration and metadata management
 
 <details>
@@ -83,7 +80,7 @@ Runs are launched by calling the `dagster api` CLI command in a separate process
 
 When runs are launched, the user code process/container streams structured metadata (containing everything that's viewable in the integrated logs viewer in the Dagster+ UI) back to Dagster+ over a well-defined GraphQL interface. Structured metadata is stored in Amazon RDS, encrypted at rest.
 
-By default, the run worker also uploads the compute logs (raw `stdout` and `stderr` from runs) to Dagster+. If you don't want to upload logs, you can disable this feature in the [agent settings](/dagster-plus/deployment/hybrid/agents/settings).
+By default, the run worker also uploads the compute logs (raw `stdout` and `stderr` from runs) to Dagster+. If you don't want to upload logs, you can disable this feature in the [agent settings](/dagster-plus/deployment/management/settings/hybrid-agent-settings).
 
 ### Ingress
 

diff --git a/...ta/docs/dagster-plus/deployment/deployment-types/hybrid/docker/configuration.md b/...ta/docs/dagster-plus/deployment/deployment-types/hybrid/docker/configuration.md
@@ -0,0 +1,7 @@
+---
+title: Docker agent configuration
+sidebar_position: 200
+unlisted: true
+---
+
+{/* TODO copy from https://docs.dagster.io/dagster-plus/deployment/agents/docker/configuration-reference */}
diff --git a/...gster-plus/deployment/migration/index.mdx → ...t/deployment-types/hybrid/docker/index.md b/...gster-plus/deployment/migration/index.mdx → ...t/deployment-types/hybrid/docker/index.md
@@ -1,11 +1,8 @@
 ---
-title: Migration
-displayed_sidebar: "dagsterPlus"
+title: Docker agent
 sidebar_position: 30
 ---
 
-# Migration
-
 import DocCardList from '@theme/DocCardList';
 
 <DocCardList />
diff --git a/.../docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/docker/setup.md b/.../docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/docker/setup.md
@@ -0,0 +1,7 @@
+---
+title: Docker agent setup
+sidebar_position: 100
+unlisted: true
+---
+
+{/* TODO copy from https://docs.dagster.io/dagster-plus/deployment/agents/docker/configuring-running-docker-agent */}
diff --git a/...s/dagster-plus/deployment/hybrid/index.md → ...ployment/deployment-types/hybrid/index.md b/...s/dagster-plus/deployment/hybrid/index.md → ...ployment/deployment-types/hybrid/index.md
@@ -1,11 +1,9 @@
 ---
 title: "Hybrid deployment"
-displayed_sidebar: "dagsterPlus"
+sidebar_label: Hybrid
 sidebar_position: 20
 ---
 
-# Hybrid deployment
-
 In a Dagster+ Hybrid deployment, the orchestration control plane is run by Dagster+ while your Dagster code is executed within your environment.
 
 [comment]: <> (TODO: Architecture diagram)
@@ -23,10 +21,10 @@ To get started with a Hybrid deployment you'll need to:
 The Dagster+ agent is a long-lived process that polls Dagster+'s API servers for new work.
 
 See the following guides for setting up an agent:
- - [Kubernetes](/dagster-plus/deployment/hybrid/agents/kubernetes)
- - [AWS ECS](/dagster-plus/deployment/hybrid/agents/amazon-ecs-new-vpc)
- - [Docker](/dagster-plus/deployment/hybrid/agents/docker)
- - [Locally](/dagster-plus/deployment/hybrid/agents/local)
+ - [Kubernetes](/dagster-plus/deployment/deployment-types/hybrid/kubernetes)
+ - [AWS ECS](/dagster-plus/deployment/deployment-types/hybrid/amazon-ecs/new-vpc)
+ - [Docker](/dagster-plus/deployment/deployment-types/hybrid/docker)
+ - [Locally](/dagster-plus/deployment/deployment-types/hybrid/local)
 
 
 ## What you'll see in your environment

diff --git a/...ocs/dagster-plus/deployment/deployment-types/hybrid/kubernetes/configuration.md b/...ocs/dagster-plus/deployment/deployment-types/hybrid/kubernetes/configuration.md
@@ -0,0 +1,7 @@
+---
+title: Kubernetes agent configuration
+sidebar_position: 200
+unlisted: true
+---
+
+{/* TODO copy from https://docs.dagster.io/dagster-plus/deployment/agents/kubernetes/configuration-reference */}
diff --git a/...s-beta/docs/dagster-plus/deployment/deployment-types/hybrid/kubernetes/index.md b/...s-beta/docs/dagster-plus/deployment/deployment-types/hybrid/kubernetes/index.md
@@ -0,0 +1,8 @@
+---
+title: Kubernetes agent
+sidebar_position: 40
+---
+
+import DocCardList from '@theme/DocCardList';
+
+<DocCardList />
diff --git a/...us/deployment/hybrid/agents/kubernetes.md → ...ployment-types/hybrid/kubernetes/setup.md b/...us/deployment/hybrid/agents/kubernetes.md → ...ployment-types/hybrid/kubernetes/setup.md
@@ -1,12 +1,8 @@
 ---
-title: "Running Dagster+ agents on Kubernetes"
-displayed_sidebar: "dagsterPlus"
-sidebar_position: 300
-sidebar_label: "Kubernetes"
+title: Kubernetes agent setup
+sidebar_position: 100
 ---
 
-# Running Dagster+ agents on Kubernetes
-
 This page provides instructions for running the [Dagster+ agent](/todo) on a [Kubernetes](https://kubernetes.io) cluster.
 
 ## Installation
@@ -28,7 +24,7 @@ kubectl create namespace dagster-cloud
 
 ### Step 2: Create an agent token secret
 
-[Generate an agent token](/dagster-plus/deployment/hybrid/tokens) and set it as a Kubernetes secret:
+[Generate an agent token](/dagster-plus/deployment/management/tokens) and set it as a Kubernetes secret:
 
 ```shell
 kubectl --namespace dagster-cloud create secret generic dagster-cloud-agent-token --from-literal=DAGSTER_CLOUD_AGENT_TOKEN=<token>
@@ -90,12 +86,12 @@ There are three places to customize how Dagster interacts with Kubernetes:
 
 Changes apply in a hierarchy, for example, a customization for an asset will override a default set globally in the agent configuration. Attributes that are not customized will use the global defaults.
 
-An exhaustive list of settings is available [here](/dagster-plus/deployment/hybrid/agents/settings), but common options are presented below.
+An exhaustive list of settings is available [here](/dagster-plus/deployment/management/settings/hybrid-agent-settings), but common options are presented below.
 
 
 ### Configure your agents to serve branch deployments
 
-[Branch deployments](/dagster-plus/features/branch-deployments/index.mdx) are lightweight staging environments created for each code change. To configure your Dagster+ agent to manage them:
+[Branch deployments](/dagster-plus/features/ci-cd/branch-deployments/index.md) are lightweight staging environments created for each code change. To configure your Dagster+ agent to manage them:
 
 ```yaml
 # values.yaml
@@ -223,7 +219,7 @@ helm --namespace dagster-cloud upgrade agent \
 
 ### Make secrets available to your code
 
-You can make secrets available [through the Dagster+ web interface](/dagster-plus/deployment/environment-variables) or through Kubernetes. Configuring secrets through Kubernetes has the benefit that Dagster+ never stores or accesses the secrets, and they can be managed as code. First, create the Kubernetes secret:
+You can make secrets available [through the Dagster+ web interface](/dagster-plus/deployment/management/environment-variables) or through Kubernetes. Configuring secrets through Kubernetes has the benefit that Dagster+ never stores or accesses the secrets, and they can be managed as code. First, create the Kubernetes secret:
 
 ```bash
 kubectl create secret generic database-password-kubernetes-secret \

diff --git a/...er-plus/deployment/hybrid/agents/local.md → ...ployment/deployment-types/hybrid/local.md b/...er-plus/deployment/hybrid/agents/local.md → ...ployment/deployment-types/hybrid/local.md
@@ -1,11 +1,10 @@
 ---
-title: "Running a Dagster+ agent locally"
-displayed_sidebar: "dagsterPlus"
-sidebar_position: 600
-sidebar_label: "Local"
+title: Running a local agent
+sidebar_position: 20
+sidebar_label: Local agent
 ---
 
-Local agents are a good way to experiment with Dagster+ before deploying a more scalable Hybrid agent like [Kubernetes](/dagster-plus/deployment/hybrid/agents/kubernetes) or [Amazon ECS](/todo).
+Local agents are a good way to experiment with Dagster+ before deploying a more scalable Hybrid agent like [Kubernetes](/dagster-plus/deployment/deployment-types/hybrid/kubernetes) or [Amazon ECS](/todo).
 
 :::note
 Local agents aren't well suited for most production use cases. If you're running the local agent in production, make sure that:

diff --git a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/multiple.md b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/multiple.md
@@ -0,0 +1,8 @@
+---
+title: Running multiple agents
+sidebar_position: 60
+sidebar_label: Multiple agents
+unlisted: true
+---
+
+{/* TODO copy from https://docs.dagster.io/dagster-plus/deployment/agents/running-multiple-agents */}
diff --git a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/index.md b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/index.md
@@ -0,0 +1,8 @@
+---
+title: Deployment types
+sidebar_position: 20
+---
+
+import DocCardList from '@theme/DocCardList';
+
+<DocCardList />
diff --git a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/multi-tenancy.md b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/multi-tenancy.md
@@ -0,0 +1,5 @@
+---
+title: Multi-tenant
+sidebar_position: 30
+unlisted: true
+---
diff --git a/...loyment/serverless/ci-cd-in-serverless.md → ...t-types/serverless/ci-cd-in-serverless.md b/...loyment/serverless/ci-cd-in-serverless.md → ...t-types/serverless/ci-cd-in-serverless.md
@@ -1,17 +1,14 @@
 ---
-title: 'CI/CD in Serverless '
-displayed_sidebar: 'dagsterPlus'
+title: 'CI/CD in Serverless'
 sidebar_position: 300
 ---
 
-# CI/CD in Serverless
-
-If you're a GitHub or GitLab user, you can use our predefined workflows to seamlessly deploy and synchronize your code to Dagster+. You can also use other Git providers or a local Git repository with our [dagster-cloud CLI](/dagster-plus/features/branch-deployments/dagster-cloud-cli) to run your own CI/CD process.
+If you're a GitHub or GitLab user, you can use our predefined workflows to seamlessly deploy and synchronize your code to Dagster+. You can also use other Git providers or a local Git repository with our [dagster-cloud CLI](/dagster-plus/deployment/management/dagster-cloud-cli) to run your own CI/CD process.
 
 <Tabs groupId="method">
 <TabItem value="GitHub" label="With GitHub">
 
-If you're a GitHub user, with a single click our GitHub app with GitHub Actions can set up a repo containing skeleton code and configuration for you consistent with Dagster+'s best practices. Pushing your code changes to the `main` branch will automatically deploy them to your `prod` Serverless deployment. Pull requests will spin up ephemeral [branch deployments](/dagster-plus/features/branch-deployments/index.mdx) that you can view in the Dagster+ UI for previewing and testing.
+If you're a GitHub user, with a single click our GitHub app with GitHub Actions can set up a repo containing skeleton code and configuration for you consistent with Dagster+'s best practices. Pushing your code changes to the `main` branch will automatically deploy them to your `prod` Serverless deployment. Pull requests will spin up ephemeral [branch deployments](/dagster-plus/features/ci-cd/branch-deployments/index.md) that you can view in the Dagster+ UI for previewing and testing.
 
 :::note
 **If you are importing a Dagster project that's in an existing GitHub repo:**
@@ -28,13 +25,13 @@ If you're a GitHub user, with a single click our GitHub app with GitHub Actions
 
 <TabItem value="GitLab" label="With GitLab">
 
-If you're a GitLab user, with a single click our GitLab app can set up a repo containing skeleton code and CI/CD configuration for you consistent with Dagster+'s best practices. Pushing your code changes to the `main` branch will automatically deploy them to your `prod` Serverless deployment. Pull requests will spin up ephemeral [branch deployments](/dagster-plus/features/branch-deployments/index.mdx) that you can view in the Dagster+ UI for previewing and testing.
+If you're a GitLab user, with a single click our GitLab app can set up a repo containing skeleton code and CI/CD configuration for you consistent with Dagster+'s best practices. Pushing your code changes to the `main` branch will automatically deploy them to your `prod` Serverless deployment. Pull requests will spin up ephemeral [branch deployments](/dagster-plus/features/ci-cd/branch-deployments/index.md) that you can view in the Dagster+ UI for previewing and testing.
 
 </TabItem>
 
 <TabItem value="Other" label="Other Git providers or local development">
 
-If you don't want to use our automated GitHub/GitLab process, we offer [the powerful `dagster-cloud` command-line interface (CLI)](/dagster-plus/features/branch-deployments/dagster-cloud-cli) that you can use in another CI environment or locally.
+If you don't want to use our automated GitHub/GitLab process, we offer [the powerful `dagster-cloud` command-line interface (CLI)](/dagster-plus/features/ci-cd/branch-deployments/dagster-cloud-cli) that you can use in another CI environment or locally.
 
 First, [create a new project](/getting-started/quickstart) with the Dagster open source CLI.
 
@@ -62,7 +59,7 @@ install_requires=[
 
 :::
 
-Next, install the [`dagster-cloud` CLI](/dagster-plus/features/branch-deployments/dagster-cloud-cli) and use its `configure` command to authenticate it to your Dagster+ organization.
+Next, install the [`dagster-cloud` CLI](/dagster-plus/features/ci-cd/branch-deployments/dagster-cloud-cli) and use its `configure` command to authenticate it to your Dagster+ organization.
 
 **Note:** The CLI requires a recent version of Python 3 and Docker.
 
@@ -71,7 +68,7 @@ pip install dagster-cloud
 dagster-cloud configure
 ```
 
-You can also configure the `dagster-cloud` tool non-interactively; see [the CLI docs](/dagster-plus/features/branch-deployments/dagster-cloud-cli) for more information.
+You can also configure the `dagster-cloud` tool non-interactively; see [the CLI docs](/dagster-plus/features/ci-cd/branch-deployments/dagster-cloud-cli) for more information.
 
 Finally, deploy your project to Dagster+ using the `serverless` command:
 

diff --git a/...plus/deployment/serverless/dagster-ips.md → ...eployment-types/serverless/dagster-ips.md b/...plus/deployment/serverless/dagster-ips.md → ...eployment-types/serverless/dagster-ips.md
@@ -1,11 +1,9 @@
 ---
-title: 'IP addresses'
-displayed_sidebar: 'dagsterPlus'
+title: Dagster+ Serverless IP addresses
+sidebar_label: IP addresses
 sidebar_position: 400
 ---
 
-# Dagster+ Serverless IP addresses
-
 Serverless code will make requests from one of the following IP addresses. You may need to whitelist / allowlist them for services your code interacts with.
 
 ```plain