dagster-io · neverett · Dec 12, 2024 · Dec 5, 2024 · Dec 5, 2024 · Dec 5, 2024
@@ -1,8 +1,7 @@
 ---
 title: 'Dagster+ code requirements'
-displayed_sidebar: 'dagsterPlus'
 sidebar_label: "Code requirements"
-sidebar_position: 100
+sidebar_position: 10
 ---
 
 Your Dagster project must meet a few requirements to run in Dagster+.
@@ -34,7 +33,7 @@ To work with Dagster+, your Dagster code:
 
 ## Hybrid deployment requirements
 
-If you're using [Hybrid Deployment](/dagster-plus/deployment/hybrid), there are a few additional requirements.
+If you're using [Hybrid Deployment](/dagster-plus/deployment/deployment-types/hybrid), there are a few additional requirements.
 
 - **If using an Amazon Elastic Container Service (ECS), Kubernetes, or Docker agent**, your code must be packaged into a Docker image and pushed to a registry your agent can access. Dagster+ doesn't need access to your image - the agent only needs to be able to pull it.
 

@@ -0,0 +1,7 @@
+---
+title: Configuration reference
+sidebar_position: 400
+unlisted: true
+---
+
+{/* TODO copy from https://docs.dagster.io/dagster-plus/deployment/agents/amazon-ecs/configuration-reference */}
@@ -0,0 +1,7 @@
+---
+title: Existing VPC setup
+sidebar_position: 200
+unlisted: true
+---
+
+{/* TODO copy from https://docs.dagster.io/dagster-plus/deployment/agents/amazon-ecs/creating-ecs-agent-existing-vpc */}
@@ -0,0 +1,8 @@
+---
+title: Amazon ECS agent
+sidebar_position: 50
+---
+
+import DocCardList from '@theme/DocCardList';
+
+<DocCardList />
@@ -0,0 +1,7 @@
+---
+title: Manual provision setup
+sidebar_position: 300
+unlisted: true
+---
+
+{/* TODO move content from https://docs.dagster.io/dagster-plus/deployment/agents/amazon-ecs/manually-provisioning-ecs-agent */}
@@ -0,0 +1,7 @@
+---
+title: New VPC setup
+sidebar_position: 100
+unlisted: true
+---
+
+{/* TODO copy from https://docs.dagster.io/dagster-plus/deployment/agents/amazon-ecs/creating-ecs-agent-new-vpc */}
@@ -0,0 +1,8 @@
+---
+title: Upgrading CloudFormation for an Amazon ECS agent
+sidebar_label: Upgrading CloudFormation
+sidebar_position: 500
+unlisted: true
+---
+
+{/* TODO move from https://docs.dagster.io/dagster-plus/deployment/agents/amazon-ecs/upgrading-cloudformation-template */}
@@ -1,11 +1,8 @@
 ---
 title: 'Dagster+ Hybrid architecture'
-displayed_sidebar: 'dagsterPlus'
-sidebar_position: 100
+sidebar_position: 10
 ---
 
-# Dagster+ Hybrid architecture
-
 The Hybrid architecture is the most flexible and secure way to deploy Dagster+. It allows you to run your user code in your environment while leveraging Dagster+'s infrastructure for orchestration and metadata management
 
 <details>
@@ -83,7 +80,7 @@ Runs are launched by calling the `dagster api` CLI command in a separate process
 
 When runs are launched, the user code process/container streams structured metadata (containing everything that's viewable in the integrated logs viewer in the Dagster+ UI) back to Dagster+ over a well-defined GraphQL interface. Structured metadata is stored in Amazon RDS, encrypted at rest.
 
-By default, the run worker also uploads the compute logs (raw `stdout` and `stderr` from runs) to Dagster+. If you don't want to upload logs, you can disable this feature in the [agent settings](/dagster-plus/deployment/hybrid/agents/settings).
+By default, the run worker also uploads the compute logs (raw `stdout` and `stderr` from runs) to Dagster+. If you don't want to upload logs, you can disable this feature in the [agent settings](/dagster-plus/deployment/management/settings/hybrid-agent-settings).
 
 ### Ingress
 

@@ -0,0 +1,7 @@
+---
+title: Docker agent configuration
+sidebar_position: 200
+unlisted: true
+---
+
+{/* TODO copy from https://docs.dagster.io/dagster-plus/deployment/agents/docker/configuration-reference */}
@@ -1,11 +1,8 @@
 ---
-title: Migration
-displayed_sidebar: "dagsterPlus"
+title: Docker agent
 sidebar_position: 30
 ---
 
-# Migration
-
 import DocCardList from '@theme/DocCardList';
 
 <DocCardList />
@@ -0,0 +1,7 @@
+---
+title: Docker agent setup
+sidebar_position: 100
+unlisted: true
+---
+
+{/* TODO copy from https://docs.dagster.io/dagster-plus/deployment/agents/docker/configuring-running-docker-agent */}
@@ -1,11 +1,9 @@
 ---
 title: "Hybrid deployment"
-displayed_sidebar: "dagsterPlus"
+sidebar_label: Hybrid
 sidebar_position: 20
 ---
 
-# Hybrid deployment
-
 In a Dagster+ Hybrid deployment, the orchestration control plane is run by Dagster+ while your Dagster code is executed within your environment.
 
 [comment]: <> (TODO: Architecture diagram)
@@ -16,24 +14,24 @@ To get started with a Hybrid deployment you'll need to:
 
 1. Create a [Dagster+ organization](https://dagster.cloud/signup)
 2. Install a Dagster+ Hybrid Agent
-3. [Add a code location](/dagster-plus/deployment/code-locations), typically using a Git repository and CI/CD
+3. [Add a code location](/dagster-plus/features/code-locations), typically using a Git repository and CI/CD
 
 ## Dagster+ Hybrid agents
 
 The Dagster+ agent is a long-lived process that polls Dagster+'s API servers for new work.
 
 See the following guides for setting up an agent:
- - [Kubernetes](/dagster-plus/deployment/hybrid/agents/kubernetes)
- - [AWS ECS](/dagster-plus/deployment/hybrid/agents/amazon-ecs-new-vpc)
- - [Docker](/dagster-plus/deployment/hybrid/agents/docker)
- - [Locally](/dagster-plus/deployment/hybrid/agents/local)
+ - [Kubernetes](/dagster-plus/deployment/deployment-types/hybrid/kubernetes)
+ - [AWS ECS](/dagster-plus/deployment/deployment-types/hybrid/amazon-ecs/new-vpc)
+ - [Docker](/dagster-plus/deployment/deployment-types/hybrid/docker)
+ - [Locally](/dagster-plus/deployment/deployment-types/hybrid/local)
 
 
 ## What you'll see in your environment
 
 ### Code location servers
 
-Dagster+ runs your Dagster projects through code locations. To get started, follow this guide for [adding a code location](/dagster-plus/deployment/code-locations).
+Dagster+ runs your Dagster projects through code locations. To get started, follow this guide for [adding a code location](/dagster-plus/features/code-locations).
 
 When you inform Dagster+ about a new code location, we enqueue instructions for your agent to launch a new code server. The agent uses your container image to launch a code server that interacts with your Dagster definitions. The agent will run one long-standing code server for each code location. Once the code server is running, the agent will send Dagster+ metadata about your Dagster definitions that Dagster+ uses to make orchestration decisions.
 

@@ -0,0 +1,7 @@
+---
+title: Kubernetes agent configuration
+sidebar_position: 200
+unlisted: true
+---
+
+{/* TODO copy from https://docs.dagster.io/dagster-plus/deployment/agents/kubernetes/configuration-reference */}
@@ -0,0 +1,8 @@
+---
+title: Kubernetes agent
+sidebar_position: 40
+---
+
+import DocCardList from '@theme/DocCardList';
+
+<DocCardList />
@@ -1,12 +1,8 @@
 ---
-title: "Running Dagster+ agents on Kubernetes"
-displayed_sidebar: "dagsterPlus"
-sidebar_position: 300
-sidebar_label: "Kubernetes"
+title: Kubernetes agent setup
+sidebar_position: 100
 ---
 
-# Running Dagster+ agents on Kubernetes
-
 This page provides instructions for running the [Dagster+ agent](/todo) on a [Kubernetes](https://kubernetes.io) cluster.
 
 ## Installation
@@ -28,7 +24,7 @@ kubectl create namespace dagster-cloud
 
 ### Step 2: Create an agent token secret
 
-[Generate an agent token](/dagster-plus/deployment/hybrid/tokens) and set it as a Kubernetes secret:
+[Generate an agent token](/dagster-plus/deployment/management/tokens) and set it as a Kubernetes secret:
 
 ```shell
 kubectl --namespace dagster-cloud create secret generic dagster-cloud-agent-token --from-literal=DAGSTER_CLOUD_AGENT_TOKEN=<token>
@@ -85,17 +81,17 @@ kubectl --namespace dagster-cloud logs -l deployment=agent
 
 There are three places to customize how Dagster interacts with Kubernetes:
 - **Per Deployment** by configuring the Dagster+ agent using [Helm values](https://artifacthub.io/packages/helm/dagster-cloud/dagster-cloud-agent?modal=values)
-- **Per Project** by configuring the `dagster_cloud.yaml` file for your [code location](/dagster-plus/deployment/code-locations)
+- **Per Project** by configuring the `dagster_cloud.yaml` file for your [code location](/dagster-plus/features/code-locations)
 - **Per Asset or Job** by adding tags to the [asset](/todo), [job](/todo), or [customizing the Kubernetes pipes invocation](/todo)
 
 Changes apply in a hierarchy, for example, a customization for an asset will override a default set globally in the agent configuration. Attributes that are not customized will use the global defaults.
 
-An exhaustive list of settings is available [here](/dagster-plus/deployment/hybrid/agents/settings), but common options are presented below.
+An exhaustive list of settings is available [here](/dagster-plus/deployment/management/settings/hybrid-agent-settings), but common options are presented below.
 
 
 ### Configure your agents to serve branch deployments
 
-[Branch deployments](/dagster-plus/features/branch-deployments/index.mdx) are lightweight staging environments created for each code change. To configure your Dagster+ agent to manage them:
+[Branch deployments](/dagster-plus/features/ci-cd/branch-deployments/index.md) are lightweight staging environments created for each code change. To configure your Dagster+ agent to manage them:
 
 ```yaml
 # values.yaml
@@ -223,7 +219,7 @@ helm --namespace dagster-cloud upgrade agent \
 
 ### Make secrets available to your code
 
-You can make secrets available [through the Dagster+ web interface](/dagster-plus/deployment/environment-variables) or through Kubernetes. Configuring secrets through Kubernetes has the benefit that Dagster+ never stores or accesses the secrets, and they can be managed as code. First, create the Kubernetes secret:
+You can make secrets available [through the Dagster+ web interface](/dagster-plus/deployment/management/environment-variables) or through Kubernetes. Configuring secrets through Kubernetes has the benefit that Dagster+ never stores or accesses the secrets, and they can be managed as code. First, create the Kubernetes secret:
 
 ```bash
 kubectl create secret generic database-password-kubernetes-secret \

@@ -1,11 +1,10 @@
 ---
-title: "Running a Dagster+ agent locally"
-displayed_sidebar: "dagsterPlus"
-sidebar_position: 600
-sidebar_label: "Local"
+title: Running a local agent
+sidebar_position: 20
+sidebar_label: Local agent
 ---
 
-Local agents are a good way to experiment with Dagster+ before deploying a more scalable Hybrid agent like [Kubernetes](/dagster-plus/deployment/hybrid/agents/kubernetes) or [Amazon ECS](/todo).
+Local agents are a good way to experiment with Dagster+ before deploying a more scalable Hybrid agent like [Kubernetes](/dagster-plus/deployment/deployment-types/hybrid/kubernetes) or [Amazon ECS](/todo).
 
 :::note
 Local agents aren't well suited for most production use cases. If you're running the local agent in production, make sure that:

@@ -0,0 +1,8 @@
+---
+title: Running multiple agents
+sidebar_position: 60
+sidebar_label: Multiple agents
+unlisted: true
+---
+
+{/* TODO copy from https://docs.dagster.io/dagster-plus/deployment/agents/running-multiple-agents */}
@@ -0,0 +1,8 @@
+---
+title: Deployment types
+sidebar_position: 20
+---
+
+import DocCardList from '@theme/DocCardList';
+
+<DocCardList />
@@ -0,0 +1,5 @@
+---
+title: Multi-tenant
+sidebar_position: 30
+unlisted: true
+---
@@ -1,17 +1,14 @@
 ---
-title: 'CI/CD in Serverless '
-displayed_sidebar: 'dagsterPlus'
+title: 'CI/CD in Serverless'
 sidebar_position: 300
 ---
 
-# CI/CD in Serverless
-
-If you're a GitHub or GitLab user, you can use our predefined workflows to seamlessly deploy and synchronize your code to Dagster+. You can also use other Git providers or a local Git repository with our [dagster-cloud CLI](/dagster-plus/features/branch-deployments/dagster-cloud-cli) to run your own CI/CD process.
+If you're a GitHub or GitLab user, you can use our predefined workflows to seamlessly deploy and synchronize your code to Dagster+. You can also use other Git providers or a local Git repository with our [dagster-cloud CLI](/dagster-plus/deployment/management/dagster-cloud-cli) to run your own CI/CD process.
 
 <Tabs groupId="method">
 <TabItem value="GitHub" label="With GitHub">
 
-If you're a GitHub user, with a single click our GitHub app with GitHub Actions can set up a repo containing skeleton code and configuration for you consistent with Dagster+'s best practices. Pushing your code changes to the `main` branch will automatically deploy them to your `prod` Serverless deployment. Pull requests will spin up ephemeral [branch deployments](/dagster-plus/features/branch-deployments/index.mdx) that you can view in the Dagster+ UI for previewing and testing.
+If you're a GitHub user, with a single click our GitHub app with GitHub Actions can set up a repo containing skeleton code and configuration for you consistent with Dagster+'s best practices. Pushing your code changes to the `main` branch will automatically deploy them to your `prod` Serverless deployment. Pull requests will spin up ephemeral [branch deployments](/dagster-plus/features/ci-cd/branch-deployments/index.md) that you can view in the Dagster+ UI for previewing and testing.
 
 :::note
 **If you are importing a Dagster project that's in an existing GitHub repo:**
@@ -28,13 +25,13 @@ If you're a GitHub user, with a single click our GitHub app with GitHub Actions
 
 <TabItem value="GitLab" label="With GitLab">
 
-If you're a GitLab user, with a single click our GitLab app can set up a repo containing skeleton code and CI/CD configuration for you consistent with Dagster+'s best practices. Pushing your code changes to the `main` branch will automatically deploy them to your `prod` Serverless deployment. Pull requests will spin up ephemeral [branch deployments](/dagster-plus/features/branch-deployments/index.mdx) that you can view in the Dagster+ UI for previewing and testing.
+If you're a GitLab user, with a single click our GitLab app can set up a repo containing skeleton code and CI/CD configuration for you consistent with Dagster+'s best practices. Pushing your code changes to the `main` branch will automatically deploy them to your `prod` Serverless deployment. Pull requests will spin up ephemeral [branch deployments](/dagster-plus/features/ci-cd/branch-deployments/index.md) that you can view in the Dagster+ UI for previewing and testing.
 
 </TabItem>
 
 <TabItem value="Other" label="Other Git providers or local development">
 
-If you don't want to use our automated GitHub/GitLab process, we offer [the powerful `dagster-cloud` command-line interface (CLI)](/dagster-plus/features/branch-deployments/dagster-cloud-cli) that you can use in another CI environment or locally.
+If you don't want to use our automated GitHub/GitLab process, we offer [the powerful `dagster-cloud` command-line interface (CLI)](/dagster-plus/features/ci-cd/branch-deployments/dagster-cloud-cli) that you can use in another CI environment or locally.
 
 First, [create a new project](/getting-started/quickstart) with the Dagster open source CLI.
 
@@ -62,7 +59,7 @@ install_requires=[
 
 :::
 
-Next, install the [`dagster-cloud` CLI](/dagster-plus/features/branch-deployments/dagster-cloud-cli) and use its `configure` command to authenticate it to your Dagster+ organization.
+Next, install the [`dagster-cloud` CLI](/dagster-plus/features/ci-cd/branch-deployments/dagster-cloud-cli) and use its `configure` command to authenticate it to your Dagster+ organization.
 
 **Note:** The CLI requires a recent version of Python 3 and Docker.
 
@@ -71,7 +68,7 @@ pip install dagster-cloud
 dagster-cloud configure
 ```
 
-You can also configure the `dagster-cloud` tool non-interactively; see [the CLI docs](/dagster-plus/features/branch-deployments/dagster-cloud-cli) for more information.
+You can also configure the `dagster-cloud` tool non-interactively; see [the CLI docs](/dagster-plus/features/ci-cd/branch-deployments/dagster-cloud-cli) for more information.
 
 Finally, deploy your project to Dagster+ using the `serverless` command:
 

@@ -1,11 +1,9 @@
 ---
-title: 'IP addresses'
-displayed_sidebar: 'dagsterPlus'
+title: Dagster+ Serverless IP addresses
+sidebar_label: IP addresses
 sidebar_position: 400
 ---
 
-# Dagster+ Serverless IP addresses
-
 Serverless code will make requests from one of the following IP addresses. You may need to whitelist / allowlist them for services your code interacts with.
 
 ```plain

@@ -1,6 +1,6 @@
 ---
 title: 'Serverless deployment'
-displayed_sidebar: 'dagsterPlus'
+sidebar_label: Serverless
 sidebar_position: 10
 ---
 
@@ -14,7 +14,7 @@ Dagster+ Serverless is a fully managed version of Dagster+ and is the easiest wa
 
 Serverless works best with workloads that primarily orchestrate other services or perform light computation. Most workloads fit into this category, especially those that orchestrate third-party SaaS products like cloud data warehouses and ETL tools.
 
-If any of the following are applicable, you should select [Hybrid deployment](/dagster-plus/deployment/hybrid):
+If any of the following are applicable, you should select [Hybrid deployment](/dagster-plus/deployment/deployment-types/hybrid):
 
 - You require substantial computational resources. For example, training a large machine learning (ML) model in-process
 - Your dataset is too large to fit in memory. For example, training a large ML model in-process on a terabyte of data