diff --git a/docs/content/api/modules.json.gz b/docs/content/api/modules.json.gz index babd11583e374..00caf49d509e0 100644 Binary files a/docs/content/api/modules.json.gz and b/docs/content/api/modules.json.gz differ diff --git a/docs/content/api/searchindex.json.gz b/docs/content/api/searchindex.json.gz index 5ca4d157c09eb..39b9eab48e8bb 100644 Binary files a/docs/content/api/searchindex.json.gz and b/docs/content/api/searchindex.json.gz differ diff --git a/docs/content/api/sections.json.gz b/docs/content/api/sections.json.gz index 1254495534a4f..c970656e88c06 100644 Binary files a/docs/content/api/sections.json.gz and b/docs/content/api/sections.json.gz differ diff --git a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/ci-cd-in-serverless.md b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/ci-cd-in-serverless.md index 265a378ea82b0..dc094da1f98fb 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/ci-cd-in-serverless.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/ci-cd-in-serverless.md @@ -83,5 +83,3 @@ dagster-cloud serverless deploy-python-executable ./my-dagster-project \ - ---- diff --git a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/index.md b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/index.md index 02b9ce40ada82..672ded0e54bbc 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/index.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/index.md @@ -8,9 +8,7 @@ sidebar_position: 10 Dagster+ Serverless is a fully managed version of Dagster+ and is the easiest way to get started with Dagster. With a Serverless deployment, you can run your Dagster jobs without spinning up any infrastructure yourself. ---- - -## When to choose Serverless \{#when-to-choose-serverless} +## Serverless vs Hybrid Serverless works best with workloads that primarily orchestrate other services or perform light computation. Most workloads fit into this category, especially those that orchestrate third-party SaaS products like cloud data warehouses and ETL tools. @@ -21,9 +19,7 @@ If any of the following are applicable, you should select [Hybrid deployment](/d - You need to distribute computation across many nodes for a single run. Dagster+ runs currently execute on a single node with 4 CPUs - You don't want to add Dagster Labs as a data processor ---- - -## Limitations \{#limitations} +## Limitations Serverless is subject to the following limitations: @@ -36,8 +32,6 @@ Serverless is subject to the following limitations: Dagster+ Pro customers may request a quota increase by [contacting Sales](https://dagster.io/contact). ---- - ## Next steps -To start using Dagster+ Serverless, follow our [Getting started with Dagster+](/dagster-plus/getting-started) guide. +To start using Dagster+ Serverless, follow the steps in [Getting started with Dagster+](/dagster-plus/getting-started). diff --git a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/run-isolation.md b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/run-isolation.md index c9c5d04ab6756..ec45953a7632f 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/run-isolation.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/run-isolation.md @@ -15,8 +15,6 @@ To follow the steps in this guide, you'll need: - An understanding of [Dagster+ deployment settings](/dagster-plus/deployment/management/settings/deployment-settings) ---- - ## Differences between isolated and non-isolated runs - [**Isolated runs**](#isolated-runs-default) execute in their own container. They're the default and are intended for production and compute-heavy use cases. diff --git a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/runtime-environment.md b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/runtime-environment.md index 750eefa59919a..ae13107a69f9b 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/runtime-environment.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/runtime-environment.md @@ -7,13 +7,13 @@ sidebar_position: 100 By default, Dagster+ Serverless will package your code as PEX files and deploys them on Docker images. Using PEX files significantly reduces the time to deploy since it does not require building a new Docker image and provisioning a new container for every code change. However you are able to customize the Serverless runtime environment in various ways: - [Add dependencies](#add-dependencies) -- [Use a different Python version](#python-version) -- [Use a different base image](#base-image) -- [Include data files](#data-files) -- [Disable PEX deploys](#disable-pex) -- [Use private Python packages](#private-packages) +- [Use a different Python version](#use-a-different-python-version) +- [Use a different base image](#use-a-different-base-image) +- [Include data files](#include-data-files) +- [Disable PEX deploys](#disable-pex-deploys) +- [Use private Python packages](#use-private-python-packages) -## Add dependencies \{#add-dependencies} +## Add dependencies You can add dependencies by including the corresponding Python libraries in your Dagster project's `setup.py` file. These should follow [PEP 508](https://peps.python.org/pep-0508/). @@ -39,9 +39,9 @@ setup( ) ``` -To add a package from a private GitHub repository, see: [Use private Python packages](#private-packages) +To add a package from a private GitHub repository, see [Use private Python packages](#use-private-python-packages) -## Use a different Python version \{#python-version} +## Use a different Python version The default Python version for Dagster+ Serverless is Python 3.9. Python versions 3.10 through 3.12 are also supported. You can specify the Python version you want to use in your GitHub or GitLab workflow, or by using the `dagster-cloud` CLI. @@ -70,7 +70,7 @@ dagster-cloud serverless deploy-python-executable --python-version=3.11 --locati -## Use a different base image \{#base-image} +## Use a different base image Dagster+ runs your code on a Docker image that we build as follows: @@ -117,7 +117,7 @@ Setting a custom base image isn't supported for GitLab CI/CD workflows out of th -## Include data files \{#data-files} +## Include data files To add data files to your deployment, use the [Data Files Support](https://setuptools.pypa.io/en/latest/userguide/datafiles.html) built into Python's `setup.py`. This requires adding a `package_data` or `include_package_data` keyword in the call to `setup()` in `setup.py`. For example, given this directory structure: @@ -134,7 +134,7 @@ To add data files to your deployment, use the [Data Files Support](https://setup If you want to include the data folder, modify your `setup.py` to add the `package_data` line: -## Disable PEX deploys \{#disable-pex} +## Disable PEX deploys You have the option to disable PEX-based deploys and deploy using a Docker image instead of PEX. You can disable PEX in your GitHub or GitLab workflow, or by using the `dagster-cloud` CLI. @@ -200,7 +200,7 @@ Setting a custom base image isn't supported for GitLab CI/CD workflows out of th -## Use private Python packages \{#private-packages} +## Use private Python packages If you use PEX deploys in your workflow (`ENABLE_FAST_DEPLOYS: 'true'`), the following steps can install a package from a private GitHub repository, e.g. `my-org/private-repo`, as a dependency: diff --git a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/security.md b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/security.md index bd1b23f2e1c6e..1996c5fd07abf 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/security.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/serverless/security.md @@ -32,8 +32,6 @@ To prevent this, you can use [another I/O manager](/guides/build/configure/io-ma You must have [boto3](https://pypi.org/project/boto3/) or `dagster-cloud[serverless]` installed as a project dependency otherwise the Dagster+ managed storage can fail and silently fall back to using the default I/O manager. ::: -## Adding environment variables and secrets \{#adding-secrets} +## Adding environment variables and secrets Often you'll need to securely access secrets from your jobs. Dagster+ supports several methods for adding secrets—refer to the [Dagster+ environment variables documentation](/dagster-plus/deployment/management/environment-variables) for more information. - ---- diff --git a/docs/docs-beta/docs/dagster-plus/getting-started.md b/docs/docs-beta/docs/dagster-plus/getting-started.md index fb4214b62e4ce..b515b01891a63 100644 --- a/docs/docs-beta/docs/dagster-plus/getting-started.md +++ b/docs/docs-beta/docs/dagster-plus/getting-started.md @@ -2,12 +2,16 @@ title: "Getting started with Dagster+" --- -First [create a Dagster+ organization](https://dagster.plus/signup). Note: you can sign up with: +To get started with Dagster+, you will need to create a Dagster+ organization and choose your deployment type (Serverless or Hybrid). + +## Create a Dagster+ organization + +First, [create a Dagster+ organization](https://dagster.plus/signup). You can sign up with: - a Google email address - a GitHub account -- a one-time email link, great if you are using a corporate email. You can setup SSO after completing these steps. +- a one-time email link (ideal if you are using a corporate email). You can set up SSO after completing these steps. -Next, pick your deployment type. Not sure? +## Choose your deployment type - [Dagster+ Serverless](/dagster-plus/deployment/deployment-types/serverless) is the easiest way to get started and is great for teams with limited DevOps support. In Dagster+ Serverless, your Dagster code is executed in Dagster+. You will need to be okay [giving Dagster+ the credentials](/dagster-plus/deployment/management/environment-variables) to connect to the tools you want to orchestrate. @@ -20,14 +24,12 @@ The remaining steps depend on your deployment type. We recommend following the steps in Dagster+ to add a new project. -![Screenshot of Dagster+ serverless NUX](/img/placeholder.svg) - -The Dagster+ on-boarding will guide you through: +The Dagster+ onboarding will guide you through: - creating a Git repository containing your Dagster code - setting up the necessary CI/CD actions to deploy that repository to Dagster+ :::tip -If you don't have any Dagster code yet, you will have the option to select an example quickstart project or import an existing dbt project +If you don't have any Dagster code yet, you can select an example project or import an existing dbt project. ::: See the guide on [adding code locations](/dagster-plus/deployment/code-locations) for details. @@ -35,12 +37,12 @@ See the guide on [adding code locations](/dagster-plus/deployment/code-locations -## Install a Dagster+ Hybrid agent +**Install a Dagster+ Hybrid agent** -Follow [these guides](/dagster-plus/deployment/deployment-types/hybrid) for installing a Dagster+ Hybrid agent. Not sure which agent to pick? We recommend using the Dagster+ Kubernetes agent in most cases. +Follow [these guides](/dagster-plus/deployment/deployment-types/hybrid) for installing a Dagster+ Hybrid agent. If you're not sure which agent to use, we recommend the [Dagster+ Kubernetes agent](/dagster-plus/deployment/deployment-types/hybrid/kubernetes/index.md) in most cases. -## Setup CI/CD +**Set up CI/CD** In most cases, your CI/CD process will be responsible for: - building your Dagster code into a Docker image diff --git a/docs/next/public/objects.inv b/docs/next/public/objects.inv index fd50d97032019..dc486e70947de 100644 Binary files a/docs/next/public/objects.inv and b/docs/next/public/objects.inv differ diff --git a/docs/sphinx/sections/api/apidocs/libraries/dagster-aws.rst b/docs/sphinx/sections/api/apidocs/libraries/dagster-aws.rst index e8774ae92ed5d..15d161197def5 100644 --- a/docs/sphinx/sections/api/apidocs/libraries/dagster-aws.rst +++ b/docs/sphinx/sections/api/apidocs/libraries/dagster-aws.rst @@ -49,6 +49,9 @@ ECS .. autoconfigurable:: dagster_aws.ecs.EcsRunLauncher :annotation: RunLauncher +.. autoconfigurable:: dagster_aws.ecs.ecs_executor + :annotation: ExecutorDefinition + Redshift -------- diff --git a/python_modules/libraries/dagster-aws/dagster_aws/ecs/executor.py b/python_modules/libraries/dagster-aws/dagster_aws/ecs/executor.py index c93979acca20c..1dc2b41761b0f 100644 --- a/python_modules/libraries/dagster-aws/dagster_aws/ecs/executor.py +++ b/python_modules/libraries/dagster-aws/dagster_aws/ecs/executor.py @@ -76,12 +76,13 @@ config_schema=_ECS_EXECUTOR_CONFIG_SCHEMA, requirements=multiple_process_executor_requirements(), ) +@experimental def ecs_executor(init_context: InitExecutorContext) -> Executor: """Executor which launches steps as ECS tasks. To use the `ecs_executor`, set it as the `executor_def` when defining a job: - .. literalinclude:: ../../../../../../python_modules/libraries/dagster-aws/dagster_aws_tests/ecs_tests/run_launcher_tests/executor_tests/test_example_executor_mode_def.py + .. literalinclude:: ../../../../../../python_modules/libraries/dagster-aws/dagster_aws_tests/ecs_tests/launcher_tests/executor_tests/test_example_executor_mode_def.py :start-after: start_marker :end-before: end_marker :language: python @@ -91,16 +92,16 @@ def ecs_executor(init_context: InitExecutorContext) -> Executor: .. code-block:: YAML execution: - config: - cpu: 1024 - memory: 2048 - ephemeral_storage: 10 - task_overrides: - containerOverrides: - - name: run - environment: - - name: MY_ENV_VAR - value: "my_value" + config: + cpu: 1024 + memory: 2048 + ephemeral_storage: 10 + task_overrides: + containerOverrides: + - name: run + environment: + - name: MY_ENV_VAR + value: "my_value" `max_concurrent` limits the number of ECS tasks that will execute concurrently for one run. By default there is no limit- it will maximally parallel as allowed by the DAG. Note that this is not a @@ -221,7 +222,8 @@ def _get_run_task_kwargs( tags = { **{tag["key"]: tag["value"] for tag in run_task_kwargs.get("tags", [])}, **{ - tag["key"]: tag["value"] for tag in run_launcher.build_ecs_tags_for_run_task(run, container_context) + tag["key"]: tag["value"] + for tag in run_launcher.build_ecs_tags_for_run_task(run, container_context) }, **step_handler_context.dagster_run.dagster_execution_info, "dagster/step-key": step_key, @@ -232,7 +234,6 @@ def _get_run_task_kwargs( { "key": key, "value": value, - } for key, value in tags.items() ] diff --git a/python_modules/libraries/dagster-aws/dagster_aws_tests/ecs_tests/launcher_tests/executor_tests/test_executor.py b/python_modules/libraries/dagster-aws/dagster_aws_tests/ecs_tests/launcher_tests/executor_tests/test_executor.py index eeb174b0f0a79..87e25aeadd598 100644 --- a/python_modules/libraries/dagster-aws/dagster_aws_tests/ecs_tests/launcher_tests/executor_tests/test_executor.py +++ b/python_modules/libraries/dagster-aws/dagster_aws_tests/ecs_tests/launcher_tests/executor_tests/test_executor.py @@ -27,7 +27,7 @@ def bar(): @op( tags={ - "ecs/cpu": "256", + "ecs/cpu": "1024", "ecs/memory": "512", } ) @@ -209,7 +209,9 @@ def test_executor_launch(instance_cm: Callable[..., ContextManager[DagsterInstan run_task_kwargs = executor._step_handler.ecs.run_task.call_args[1] # type: ignore # noqa: SLF001 - breakpoint() + # resources should come from step tags + assert run_task_kwargs["overrides"]["cpu"] == "1024" + assert run_task_kwargs["overrides"]["memory"] == "512" tags = run_task_kwargs["tags"]