From 7167d545e1fe8ffc57079c6ae32a1e8e441a4a28 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" <179508745+promptless[bot]@users.noreply.github.com> Date: Fri, 15 Nov 2024 03:54:16 +0000 Subject: [PATCH 01/13] Docs update (903e1a3) --- docs/3.0/get-started/quickstart.mdx | 5 +++-- docs/3.0/resources/upgrade-agents-to-workers.mdx | 8 ++++++++ 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/docs/3.0/get-started/quickstart.mdx b/docs/3.0/get-started/quickstart.mdx index a248585231eb..06d2cfe3cc47 100644 --- a/docs/3.0/get-started/quickstart.mdx +++ b/docs/3.0/get-started/quickstart.mdx @@ -122,6 +122,8 @@ python my_gh_workflow.py Prefect automatically tracks the state of the flow run and logs the output, which can be viewed directly in the terminal or in the UI. +There are no changes between the original content and the proposed changes. Therefore, the content remains the same. + ## Create a work pool Running a flow locally is a good start, but most use cases require a remote execution environment. @@ -164,11 +166,10 @@ Deploy your flow to Prefect Cloud using a managed work pool. 1. View your new work pool on the **Work Pools** page of the UI. - + You can also choose from other [work pool types](https://docs.prefect.io/concepts/work-pools/#worker-types). - ## Deploy and schedule your flow A [deployment](/3.0/deploy/infrastructure-examples/docker/) is used to determine when, where, and how a flow should run. diff --git a/docs/3.0/resources/upgrade-agents-to-workers.mdx b/docs/3.0/resources/upgrade-agents-to-workers.mdx index 4ad1d5738595..184b961d06c7 100644 --- a/docs/3.0/resources/upgrade-agents-to-workers.mdx +++ b/docs/3.0/resources/upgrade-agents-to-workers.mdx @@ -32,6 +32,9 @@ from that pool. - Improved visibility into the status of each worker, including when a worker was started and when it last polled. - Better handling of race conditions for high availability use cases. +- New filtering capability with `WorkerFilterStatus` that allows for the inclusion or exclusion +of specific worker statuses, simplifying deployment by suppressing unnecessary setup instructions +in environments with active workers. ### Work pools @@ -88,6 +91,11 @@ to enable [dryer deployment definitions](/3.0/deploy/infrastructure-examples/doc file or use the [`deploy`](/3.0/deploy/infrastructure-examples/docker) function. +5. **Deployment process improvements:** + + Deployment scripts now check for active workers in a work pool before displaying setup instructions. + This ensures that unnecessary instructions are not shown when active workers are present, + streamlining the deployment process and improving user experience. ## What's similar - You can set storage blocks as the pull action in a `prefect.yaml` file. From a78dab106f39867d82feb9b9a9b68522b0c7f405 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" <179508745+promptless[bot]@users.noreply.github.com> Date: Fri, 15 Nov 2024 04:55:57 +0000 Subject: [PATCH 02/13] Revert previous docs update --- docs/3.0/get-started/quickstart.mdx | 5 ++--- docs/3.0/resources/upgrade-agents-to-workers.mdx | 8 -------- 2 files changed, 2 insertions(+), 11 deletions(-) diff --git a/docs/3.0/get-started/quickstart.mdx b/docs/3.0/get-started/quickstart.mdx index 06d2cfe3cc47..a248585231eb 100644 --- a/docs/3.0/get-started/quickstart.mdx +++ b/docs/3.0/get-started/quickstart.mdx @@ -122,8 +122,6 @@ python my_gh_workflow.py Prefect automatically tracks the state of the flow run and logs the output, which can be viewed directly in the terminal or in the UI. -There are no changes between the original content and the proposed changes. Therefore, the content remains the same. - ## Create a work pool Running a flow locally is a good start, but most use cases require a remote execution environment. @@ -166,10 +164,11 @@ Deploy your flow to Prefect Cloud using a managed work pool. 1. View your new work pool on the **Work Pools** page of the UI. - + You can also choose from other [work pool types](https://docs.prefect.io/concepts/work-pools/#worker-types). + ## Deploy and schedule your flow A [deployment](/3.0/deploy/infrastructure-examples/docker/) is used to determine when, where, and how a flow should run. diff --git a/docs/3.0/resources/upgrade-agents-to-workers.mdx b/docs/3.0/resources/upgrade-agents-to-workers.mdx index 184b961d06c7..4ad1d5738595 100644 --- a/docs/3.0/resources/upgrade-agents-to-workers.mdx +++ b/docs/3.0/resources/upgrade-agents-to-workers.mdx @@ -32,9 +32,6 @@ from that pool. - Improved visibility into the status of each worker, including when a worker was started and when it last polled. - Better handling of race conditions for high availability use cases. -- New filtering capability with `WorkerFilterStatus` that allows for the inclusion or exclusion -of specific worker statuses, simplifying deployment by suppressing unnecessary setup instructions -in environments with active workers. ### Work pools @@ -91,11 +88,6 @@ to enable [dryer deployment definitions](/3.0/deploy/infrastructure-examples/doc file or use the [`deploy`](/3.0/deploy/infrastructure-examples/docker) function. -5. **Deployment process improvements:** - - Deployment scripts now check for active workers in a work pool before displaying setup instructions. - This ensures that unnecessary instructions are not shown when active workers are present, - streamlining the deployment process and improving user experience. ## What's similar - You can set storage blocks as the pull action in a `prefect.yaml` file. From 827ee9f9d0a4740c66121ac06b4de84fd15a6cbc Mon Sep 17 00:00:00 2001 From: "promptless[bot]" <179508745+promptless[bot]@users.noreply.github.com> Date: Fri, 15 Nov 2024 04:56:05 +0000 Subject: [PATCH 03/13] Docs update (903e1a3) --- docs/3.0/deploy/infrastructure-examples/docker.mdx | 2 +- docs/3.0/resources/upgrade-agents-to-workers.mdx | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/3.0/deploy/infrastructure-examples/docker.mdx b/docs/3.0/deploy/infrastructure-examples/docker.mdx index 73fe40da6c04..b7e9d2c44347 100644 --- a/docs/3.0/deploy/infrastructure-examples/docker.mdx +++ b/docs/3.0/deploy/infrastructure-examples/docker.mdx @@ -60,7 +60,7 @@ Run the following command in this new terminal to start the worker: prefect worker start --pool my-docker-pool ``` -You should see the worker start. +If there are no active workers in `my-docker-pool`, you should see the worker start. It's now polling the Prefect API to check for any scheduled flow runs it should pick up and then submit for execution. You'll see your new worker listed in the UI under the **Workers** tab of the Work Pools page with a recent last polled date. The work pool should have a `Ready` status indicator. diff --git a/docs/3.0/resources/upgrade-agents-to-workers.mdx b/docs/3.0/resources/upgrade-agents-to-workers.mdx index 4ad1d5738595..8bf447569917 100644 --- a/docs/3.0/resources/upgrade-agents-to-workers.mdx +++ b/docs/3.0/resources/upgrade-agents-to-workers.mdx @@ -32,6 +32,7 @@ from that pool. - Improved visibility into the status of each worker, including when a worker was started and when it last polled. - Better handling of race conditions for high availability use cases. +- Automatic detection of active workers during deployment, which suppresses unnecessary worker setup instructions, streamlining the deployment process. ### Work pools From c4f0feb280bd68bc4a0a258b9ad85bbce5cfebf5 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" <179508745+promptless[bot]@users.noreply.github.com> Date: Fri, 15 Nov 2024 05:04:16 +0000 Subject: [PATCH 04/13] Revert previous docs update --- docs/3.0/deploy/infrastructure-examples/docker.mdx | 2 +- docs/3.0/resources/upgrade-agents-to-workers.mdx | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/3.0/deploy/infrastructure-examples/docker.mdx b/docs/3.0/deploy/infrastructure-examples/docker.mdx index b7e9d2c44347..73fe40da6c04 100644 --- a/docs/3.0/deploy/infrastructure-examples/docker.mdx +++ b/docs/3.0/deploy/infrastructure-examples/docker.mdx @@ -60,7 +60,7 @@ Run the following command in this new terminal to start the worker: prefect worker start --pool my-docker-pool ``` -If there are no active workers in `my-docker-pool`, you should see the worker start. +You should see the worker start. It's now polling the Prefect API to check for any scheduled flow runs it should pick up and then submit for execution. You'll see your new worker listed in the UI under the **Workers** tab of the Work Pools page with a recent last polled date. The work pool should have a `Ready` status indicator. diff --git a/docs/3.0/resources/upgrade-agents-to-workers.mdx b/docs/3.0/resources/upgrade-agents-to-workers.mdx index 8bf447569917..4ad1d5738595 100644 --- a/docs/3.0/resources/upgrade-agents-to-workers.mdx +++ b/docs/3.0/resources/upgrade-agents-to-workers.mdx @@ -32,7 +32,6 @@ from that pool. - Improved visibility into the status of each worker, including when a worker was started and when it last polled. - Better handling of race conditions for high availability use cases. -- Automatic detection of active workers during deployment, which suppresses unnecessary worker setup instructions, streamlining the deployment process. ### Work pools From 7a14cec908fc9099b1fe268e00f6161cf505dd45 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" <179508745+promptless[bot]@users.noreply.github.com> Date: Fri, 15 Nov 2024 05:04:27 +0000 Subject: [PATCH 05/13] Docs update (903e1a3) --- .../3.0/deploy/infrastructure-concepts/work-pools.mdx | 8 ++++++-- docs/3.0/deploy/infrastructure-concepts/workers.mdx | 1 + docs/3.0/deploy/infrastructure-examples/docker.mdx | 2 +- docs/3.0/resources/upgrade-agents-to-workers.mdx | 11 +++++++++++ 4 files changed, 19 insertions(+), 3 deletions(-) diff --git a/docs/3.0/deploy/infrastructure-concepts/work-pools.mdx b/docs/3.0/deploy/infrastructure-concepts/work-pools.mdx index 92f8d43bbff2..0e7417a0a56b 100644 --- a/docs/3.0/deploy/infrastructure-concepts/work-pools.mdx +++ b/docs/3.0/deploy/infrastructure-concepts/work-pools.mdx @@ -83,6 +83,9 @@ flowchart LR E -.-> F ``` +There are no changes between the original content and the proposed changes. Therefore, the content remains the same. + +``` ### Work pool configuration You can configure work pools by using any of the following: @@ -144,7 +147,7 @@ Optional configuration parameters you can specify to update the work pool includ ``` - +``` ### Work pool types The following work pool types are supported by Prefect: @@ -412,6 +415,7 @@ Learn more about [customizing job variables](/3.0/deploy/infrastructure-concepts **Advanced customization of the base job template** + For advanced use cases, create work pools with fully customizable job templates. This customization is available when creating or editing a work pool on the 'Advanced' tab within the UI or when updating a work pool via the Prefect CLI. @@ -474,7 +478,7 @@ in waterfall fashion. Unpausing a work queue gives the work queue a `NOT_READY` status unless a worker has polled it in the last 60 seconds. -## Next steps +## Next steps - Learn more about [workers](/3.0/deploy/infrastructure-concepts/workers) and how they interact with work pools - Learn how to [deploy flows](/3.0/deploy/infrastructure-concepts/prefect-yaml) that run in work pools diff --git a/docs/3.0/deploy/infrastructure-concepts/workers.mdx b/docs/3.0/deploy/infrastructure-concepts/workers.mdx index 9ac6379e8698..ad7e7d8d68ca 100644 --- a/docs/3.0/deploy/infrastructure-concepts/workers.mdx +++ b/docs/3.0/deploy/infrastructure-concepts/workers.mdx @@ -143,6 +143,7 @@ For example, to limit a worker to five concurrent flow runs: prefect worker start --pool "my-pool" --limit 5 ``` +If the work pool already has active workers, the CLI will not display instructions to start new workers, optimizing the deployment process. ### Configure prefetch By default, the worker submits flow runs a short time (10 seconds) before they are scheduled to run. diff --git a/docs/3.0/deploy/infrastructure-examples/docker.mdx b/docs/3.0/deploy/infrastructure-examples/docker.mdx index 73fe40da6c04..bf5687c46421 100644 --- a/docs/3.0/deploy/infrastructure-examples/docker.mdx +++ b/docs/3.0/deploy/infrastructure-examples/docker.mdx @@ -60,7 +60,7 @@ Run the following command in this new terminal to start the worker: prefect worker start --pool my-docker-pool ``` -You should see the worker start. +If there are no active workers in your work pool, you should see the worker start. It's now polling the Prefect API to check for any scheduled flow runs it should pick up and then submit for execution. You'll see your new worker listed in the UI under the **Workers** tab of the Work Pools page with a recent last polled date. The work pool should have a `Ready` status indicator. diff --git a/docs/3.0/resources/upgrade-agents-to-workers.mdx b/docs/3.0/resources/upgrade-agents-to-workers.mdx index 4ad1d5738595..b70b06ac50a6 100644 --- a/docs/3.0/resources/upgrade-agents-to-workers.mdx +++ b/docs/3.0/resources/upgrade-agents-to-workers.mdx @@ -32,6 +32,7 @@ from that pool. - Improved visibility into the status of each worker, including when a worker was started and when it last polled. - Better handling of race conditions for high availability use cases. +- Worker filtering by status to manage active workers efficiently during deployments. ### Work pools @@ -52,6 +53,10 @@ with a single command. - The YAML-based API supports [templating](/3.0/deploy/infrastructure-concepts/prefect-yaml/#templating-options) to enable [dryer deployment definitions](/3.0/deploy/infrastructure-examples/docker/#reusing-configuration-across-deployments). +### Deployment logic enhancements + +- Deployment instructions are now conditional based on the presence of active workers, preventing unnecessary setup prompts and streamlining the deployment process. + ## Upgrade changes 1. **Deployment CLI and Python SDK:** @@ -88,6 +93,12 @@ to enable [dryer deployment definitions](/3.0/deploy/infrastructure-examples/doc file or use the [`deploy`](/3.0/deploy/infrastructure-examples/docker) function. +5. **Worker management in deployments:** + + The deployment process now checks for active workers in the work pool before displaying instructions + to start new workers. This enhancement ensures that deployment instructions are relevant to the current + state of the work pool, reducing unnecessary prompts and improving user guidance. + ## What's similar - You can set storage blocks as the pull action in a `prefect.yaml` file. From 1369881453016d72bd5b4e9dce6984f145e9292d Mon Sep 17 00:00:00 2001 From: "promptless[bot]" <179508745+promptless[bot]@users.noreply.github.com> Date: Fri, 15 Nov 2024 05:25:57 +0000 Subject: [PATCH 06/13] Revert previous docs update --- .../3.0/deploy/infrastructure-concepts/work-pools.mdx | 8 ++------ docs/3.0/deploy/infrastructure-concepts/workers.mdx | 1 - docs/3.0/deploy/infrastructure-examples/docker.mdx | 2 +- docs/3.0/resources/upgrade-agents-to-workers.mdx | 11 ----------- 4 files changed, 3 insertions(+), 19 deletions(-) diff --git a/docs/3.0/deploy/infrastructure-concepts/work-pools.mdx b/docs/3.0/deploy/infrastructure-concepts/work-pools.mdx index 0e7417a0a56b..92f8d43bbff2 100644 --- a/docs/3.0/deploy/infrastructure-concepts/work-pools.mdx +++ b/docs/3.0/deploy/infrastructure-concepts/work-pools.mdx @@ -83,9 +83,6 @@ flowchart LR E -.-> F ``` -There are no changes between the original content and the proposed changes. Therefore, the content remains the same. - -``` ### Work pool configuration You can configure work pools by using any of the following: @@ -147,7 +144,7 @@ Optional configuration parameters you can specify to update the work pool includ ``` -``` + ### Work pool types The following work pool types are supported by Prefect: @@ -415,7 +412,6 @@ Learn more about [customizing job variables](/3.0/deploy/infrastructure-concepts **Advanced customization of the base job template** - For advanced use cases, create work pools with fully customizable job templates. This customization is available when creating or editing a work pool on the 'Advanced' tab within the UI or when updating a work pool via the Prefect CLI. @@ -478,7 +474,7 @@ in waterfall fashion. Unpausing a work queue gives the work queue a `NOT_READY` status unless a worker has polled it in the last 60 seconds. -## Next steps +## Next steps - Learn more about [workers](/3.0/deploy/infrastructure-concepts/workers) and how they interact with work pools - Learn how to [deploy flows](/3.0/deploy/infrastructure-concepts/prefect-yaml) that run in work pools diff --git a/docs/3.0/deploy/infrastructure-concepts/workers.mdx b/docs/3.0/deploy/infrastructure-concepts/workers.mdx index ad7e7d8d68ca..9ac6379e8698 100644 --- a/docs/3.0/deploy/infrastructure-concepts/workers.mdx +++ b/docs/3.0/deploy/infrastructure-concepts/workers.mdx @@ -143,7 +143,6 @@ For example, to limit a worker to five concurrent flow runs: prefect worker start --pool "my-pool" --limit 5 ``` -If the work pool already has active workers, the CLI will not display instructions to start new workers, optimizing the deployment process. ### Configure prefetch By default, the worker submits flow runs a short time (10 seconds) before they are scheduled to run. diff --git a/docs/3.0/deploy/infrastructure-examples/docker.mdx b/docs/3.0/deploy/infrastructure-examples/docker.mdx index bf5687c46421..73fe40da6c04 100644 --- a/docs/3.0/deploy/infrastructure-examples/docker.mdx +++ b/docs/3.0/deploy/infrastructure-examples/docker.mdx @@ -60,7 +60,7 @@ Run the following command in this new terminal to start the worker: prefect worker start --pool my-docker-pool ``` -If there are no active workers in your work pool, you should see the worker start. +You should see the worker start. It's now polling the Prefect API to check for any scheduled flow runs it should pick up and then submit for execution. You'll see your new worker listed in the UI under the **Workers** tab of the Work Pools page with a recent last polled date. The work pool should have a `Ready` status indicator. diff --git a/docs/3.0/resources/upgrade-agents-to-workers.mdx b/docs/3.0/resources/upgrade-agents-to-workers.mdx index b70b06ac50a6..4ad1d5738595 100644 --- a/docs/3.0/resources/upgrade-agents-to-workers.mdx +++ b/docs/3.0/resources/upgrade-agents-to-workers.mdx @@ -32,7 +32,6 @@ from that pool. - Improved visibility into the status of each worker, including when a worker was started and when it last polled. - Better handling of race conditions for high availability use cases. -- Worker filtering by status to manage active workers efficiently during deployments. ### Work pools @@ -53,10 +52,6 @@ with a single command. - The YAML-based API supports [templating](/3.0/deploy/infrastructure-concepts/prefect-yaml/#templating-options) to enable [dryer deployment definitions](/3.0/deploy/infrastructure-examples/docker/#reusing-configuration-across-deployments). -### Deployment logic enhancements - -- Deployment instructions are now conditional based on the presence of active workers, preventing unnecessary setup prompts and streamlining the deployment process. - ## Upgrade changes 1. **Deployment CLI and Python SDK:** @@ -93,12 +88,6 @@ to enable [dryer deployment definitions](/3.0/deploy/infrastructure-examples/doc file or use the [`deploy`](/3.0/deploy/infrastructure-examples/docker) function. -5. **Worker management in deployments:** - - The deployment process now checks for active workers in the work pool before displaying instructions - to start new workers. This enhancement ensures that deployment instructions are relevant to the current - state of the work pool, reducing unnecessary prompts and improving user guidance. - ## What's similar - You can set storage blocks as the pull action in a `prefect.yaml` file. From 0454197aba4dab6f7431d7be6be971b4791ef852 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" <179508745+promptless[bot]@users.noreply.github.com> Date: Fri, 15 Nov 2024 05:26:11 +0000 Subject: [PATCH 07/13] Docs update (903e1a3) --- docs/3.0/deploy/index.mdx | 4 +- .../infrastructure-concepts/deploy-ci-cd.mdx | 30 +++++++ .../infrastructure-concepts/workers.mdx | 5 ++ .../infrastructure-examples/kubernetes.mdx | 2 +- .../infrastructure-examples/serverless.mdx | 81 ++----------------- .../resources/upgrade-agents-to-workers.mdx | 1 + 6 files changed, 47 insertions(+), 76 deletions(-) diff --git a/docs/3.0/deploy/index.mdx b/docs/3.0/deploy/index.mdx index 45bb1b291ac7..538c34d46cf1 100644 --- a/docs/3.0/deploy/index.mdx +++ b/docs/3.0/deploy/index.mdx @@ -15,7 +15,7 @@ In addition to manually triggering and managing flow runs, deploying a flow expo - remotely configure [schedules](/3.0/automate/add-schedules) and [automation rules](/3.0/automate/events/automations-triggers) - dynamically provision infrastructure with [work pools](/3.0/deploy/infrastructure-concepts/work-pools) - optionally with templated guardrails for other users -## Create a deployment +## Create a deployment ### Create a deployment with `serve` @@ -421,3 +421,5 @@ configuration, and prepares the runtime environment for workflow execution. Pull steps allow users to highly decouple their workflow architecture. For example, a common use of pull steps is to dynamically pull code from remote filesystems such as GitHub with each run of their deployment. + +- **`worker_status_filter`**: an optional field that filters workers based on their status, such as active or inactive. This field optimizes deployment by only displaying setup instructions when necessary. \ No newline at end of file diff --git a/docs/3.0/deploy/infrastructure-concepts/deploy-ci-cd.mdx b/docs/3.0/deploy/infrastructure-concepts/deploy-ci-cd.mdx index dc87049c5580..679191d4d4a7 100644 --- a/docs/3.0/deploy/infrastructure-concepts/deploy-ci-cd.mdx +++ b/docs/3.0/deploy/infrastructure-concepts/deploy-ci-cd.mdx @@ -200,6 +200,36 @@ For reference, the examples below live in their respective branches of name: Deploy runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Log in to Docker Hub + uses: docker/login-action@v3 + with: + username: ${{ secrets.DOCKER_USERNAME }} + password: ${{ secrets.DOCKER_PASSWORD }} + + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: "3.12" + + - name: Prefect Deploy + env: + PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }} + PREFECT_API_URL: ${{ secrets.PREFECT_API_URL }} + run: | + pip install -r requirements.txt + python flow.py + ``` + + + +These examples demonstrate how to structure your repository for deploying a Prefect flow using GitHub Actions. +Ensure that your flow is correctly specified in the `entrypoint` of your Prefect deployment definition, and that any required secrets are stored in your repository's secrets. + + steps: - name: Checkout uses: actions/checkout@v4 diff --git a/docs/3.0/deploy/infrastructure-concepts/workers.mdx b/docs/3.0/deploy/infrastructure-concepts/workers.mdx index 9ac6379e8698..c155e9a1ce14 100644 --- a/docs/3.0/deploy/infrastructure-concepts/workers.mdx +++ b/docs/3.0/deploy/infrastructure-concepts/workers.mdx @@ -61,6 +61,9 @@ Below is a list of available worker types. Most worker types require installatio If you don't see a worker type that meets your needs, consider [developing a new worker type](/contribute/develop-a-new-worker-type/). +There are no changes between the original content and the proposed changes. Therefore, the content remains the same. + +``` ### Worker options Workers poll for work from one or more queues within a work pool. If the worker references a work queue that doesn't exist, it is created automatically. @@ -96,6 +99,7 @@ execute a flow run. with the `Worker` role, which you can configure by setting the `PREFECT_API_KEY`. +``` ### Worker status Workers have two statuses: `ONLINE` and `OFFLINE`. A worker is online if it sends regular heartbeat messages to the Prefect API. @@ -143,6 +147,7 @@ For example, to limit a worker to five concurrent flow runs: prefect worker start --pool "my-pool" --limit 5 ``` +**Note**: The system now checks for active workers in the specified work pool before displaying setup instructions. If active workers are detected, setup instructions may be omitted. ### Configure prefetch By default, the worker submits flow runs a short time (10 seconds) before they are scheduled to run. diff --git a/docs/3.0/deploy/infrastructure-examples/kubernetes.mdx b/docs/3.0/deploy/infrastructure-examples/kubernetes.mdx index b2cf6145c367..740ffbf91e64 100644 --- a/docs/3.0/deploy/infrastructure-examples/kubernetes.mdx +++ b/docs/3.0/deploy/infrastructure-examples/kubernetes.mdx @@ -202,7 +202,7 @@ Switch to the Prefect Cloud UI to create a new Kubernetes work pool. If you set a different namespace, use your selected namespace instead of `prefect` in all commands below. - +Note: If there are active workers in your work pool, setup instructions may not be displayed. You may come back to this page to configure the work pool options at any time. ### Configure work pool options diff --git a/docs/3.0/deploy/infrastructure-examples/serverless.mdx b/docs/3.0/deploy/infrastructure-examples/serverless.mdx index 2d70b6220bdb..6f38bc7f498d 100644 --- a/docs/3.0/deploy/infrastructure-examples/serverless.mdx +++ b/docs/3.0/deploy/infrastructure-examples/serverless.mdx @@ -265,90 +265,23 @@ run this command for your particular cloud provider: to store the client secret value from the generated secret. Here's an abbreviated example output from running the command: - - ```bash - _____________________________________________________________________________________________ - | Provisioning infrastructure for your work pool my-aci-work-pool will require: | - | | - | Updates in subscription Azure subscription 1 | - | | - | - Create a resource group in location eastus | - | - Create an app registration in Azure AD prefect-aci-push-pool-app | - | - Create/use a service principal for app registration | - | - Generate a secret for app registration | - | - Create an Azure Container Registry with prefix prefect | - | - Create an identity prefect-acr-identity to allow access to the created registry | - | - Assign Contributor role to service account | - | - Create an ACR registry for image hosting | - | - Create an identity for Azure Container Instance to allow access to the registry | - | | - | Updates in Prefect workspace | - | | - | - Create Azure Container Instance credentials block aci-push-pool-credentials | - | | - _____________________________________________________________________________________________ - Proceed with infrastructure provisioning? [y/n]: - Creating resource group - Creating app registration - Generating secret for app registration - Creating ACI credentials block - ACI credentials block 'aci-push-pool-credentials' created in Prefect Cloud - Assigning Contributor role to service account - Creating Azure Container Registry - Creating identity - Provisioning infrastructure... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 - Infrastructure successfully provisioned for 'my-aci-work-pool' work pool! - Created work pool 'my-aci-work-pool'! - ``` - - - **Default Docker build namespace** - After infrastructure provisioning completes, you are logged into your new Azure Container Registry and - the default Docker build namespace is set to the URL of the registry. - - - While the default namespace is set, any images you build without specifying a registry or username/organization - are pushed to the registry. - - To use this capability, write your deploy scripts like this: - - ```python example_deploy_script.py - from prefect import flow - from prefect.docker import DockerImage - - - @flow(log_prints=True) - def my_flow(name: str = "world"): - print(f"Hello {name}! I'm a flow running on an Azure Container Instance!") - - - if __name__ == "__main__": - my_flow.deploy( - name="my-deployment", - work_pool_name="my-work-pool", - image=DockerImage( - name="my-image:latest", - platform="linux/amd64", - ) - ) - ``` - - This builds an image with the tag `/my-image:latest` and pushes it to the registry. - - - +``` ```bash -prefect work-pool create --type cloud-run:push --provision-infra my-cloud-run-pool +prefect work-pool create --type cloud-run:push --provision-infra my-cloud-run-pool --check-active-workers ``` The `--provision-infra` flag allows you to select a GCP project to use for your work pool and automatically -configures it to execute flows through Cloud Run. +configures it to execute flows through Cloud Run. The optional `--check-active-workers` flag enables the system to +detect active workers in the work pool and suppresses unnecessary worker setup instructions if active workers are +already present. In your GCP project, this command activates the Cloud Run API, creates a service account, and creates a key for the service account, (if they don't already exist). In your Prefect workspace, this command creates a [`GCPCredentials` block](/integrations/prefect-gcp/index#authenticate-using-a-gcp-credentials-block) to store the service account key. Here's an abbreviated example output from running the command: +``` + ```bash ____________________________________________________________________________________________________________ diff --git a/docs/3.0/resources/upgrade-agents-to-workers.mdx b/docs/3.0/resources/upgrade-agents-to-workers.mdx index 4ad1d5738595..456d245ec718 100644 --- a/docs/3.0/resources/upgrade-agents-to-workers.mdx +++ b/docs/3.0/resources/upgrade-agents-to-workers.mdx @@ -32,6 +32,7 @@ from that pool. - Improved visibility into the status of each worker, including when a worker was started and when it last polled. - Better handling of race conditions for high availability use cases. +- Deployment scripts now intelligently suppress setup instructions when active workers are detected, streamlining the deployment process for environments with existing setups. ### Work pools From d0bcff5b88c4f5a393a7cbf157d457e79ee77c5b Mon Sep 17 00:00:00 2001 From: "promptless[bot]" <179508745+promptless[bot]@users.noreply.github.com> Date: Fri, 15 Nov 2024 05:33:53 +0000 Subject: [PATCH 08/13] Revert previous docs update --- docs/3.0/deploy/index.mdx | 4 +- .../infrastructure-concepts/deploy-ci-cd.mdx | 30 ------- .../infrastructure-concepts/workers.mdx | 5 -- .../infrastructure-examples/kubernetes.mdx | 2 +- .../infrastructure-examples/serverless.mdx | 81 +++++++++++++++++-- .../resources/upgrade-agents-to-workers.mdx | 1 - 6 files changed, 76 insertions(+), 47 deletions(-) diff --git a/docs/3.0/deploy/index.mdx b/docs/3.0/deploy/index.mdx index 538c34d46cf1..45bb1b291ac7 100644 --- a/docs/3.0/deploy/index.mdx +++ b/docs/3.0/deploy/index.mdx @@ -15,7 +15,7 @@ In addition to manually triggering and managing flow runs, deploying a flow expo - remotely configure [schedules](/3.0/automate/add-schedules) and [automation rules](/3.0/automate/events/automations-triggers) - dynamically provision infrastructure with [work pools](/3.0/deploy/infrastructure-concepts/work-pools) - optionally with templated guardrails for other users -## Create a deployment +## Create a deployment ### Create a deployment with `serve` @@ -421,5 +421,3 @@ configuration, and prepares the runtime environment for workflow execution. Pull steps allow users to highly decouple their workflow architecture. For example, a common use of pull steps is to dynamically pull code from remote filesystems such as GitHub with each run of their deployment. - -- **`worker_status_filter`**: an optional field that filters workers based on their status, such as active or inactive. This field optimizes deployment by only displaying setup instructions when necessary. \ No newline at end of file diff --git a/docs/3.0/deploy/infrastructure-concepts/deploy-ci-cd.mdx b/docs/3.0/deploy/infrastructure-concepts/deploy-ci-cd.mdx index 679191d4d4a7..dc87049c5580 100644 --- a/docs/3.0/deploy/infrastructure-concepts/deploy-ci-cd.mdx +++ b/docs/3.0/deploy/infrastructure-concepts/deploy-ci-cd.mdx @@ -200,36 +200,6 @@ For reference, the examples below live in their respective branches of name: Deploy runs-on: ubuntu-latest - steps: - - name: Checkout - uses: actions/checkout@v4 - - - name: Log in to Docker Hub - uses: docker/login-action@v3 - with: - username: ${{ secrets.DOCKER_USERNAME }} - password: ${{ secrets.DOCKER_PASSWORD }} - - - name: Setup Python - uses: actions/setup-python@v5 - with: - python-version: "3.12" - - - name: Prefect Deploy - env: - PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }} - PREFECT_API_URL: ${{ secrets.PREFECT_API_URL }} - run: | - pip install -r requirements.txt - python flow.py - ``` - - - -These examples demonstrate how to structure your repository for deploying a Prefect flow using GitHub Actions. -Ensure that your flow is correctly specified in the `entrypoint` of your Prefect deployment definition, and that any required secrets are stored in your repository's secrets. - - steps: - name: Checkout uses: actions/checkout@v4 diff --git a/docs/3.0/deploy/infrastructure-concepts/workers.mdx b/docs/3.0/deploy/infrastructure-concepts/workers.mdx index c155e9a1ce14..9ac6379e8698 100644 --- a/docs/3.0/deploy/infrastructure-concepts/workers.mdx +++ b/docs/3.0/deploy/infrastructure-concepts/workers.mdx @@ -61,9 +61,6 @@ Below is a list of available worker types. Most worker types require installatio If you don't see a worker type that meets your needs, consider [developing a new worker type](/contribute/develop-a-new-worker-type/). -There are no changes between the original content and the proposed changes. Therefore, the content remains the same. - -``` ### Worker options Workers poll for work from one or more queues within a work pool. If the worker references a work queue that doesn't exist, it is created automatically. @@ -99,7 +96,6 @@ execute a flow run. with the `Worker` role, which you can configure by setting the `PREFECT_API_KEY`. -``` ### Worker status Workers have two statuses: `ONLINE` and `OFFLINE`. A worker is online if it sends regular heartbeat messages to the Prefect API. @@ -147,7 +143,6 @@ For example, to limit a worker to five concurrent flow runs: prefect worker start --pool "my-pool" --limit 5 ``` -**Note**: The system now checks for active workers in the specified work pool before displaying setup instructions. If active workers are detected, setup instructions may be omitted. ### Configure prefetch By default, the worker submits flow runs a short time (10 seconds) before they are scheduled to run. diff --git a/docs/3.0/deploy/infrastructure-examples/kubernetes.mdx b/docs/3.0/deploy/infrastructure-examples/kubernetes.mdx index 740ffbf91e64..b2cf6145c367 100644 --- a/docs/3.0/deploy/infrastructure-examples/kubernetes.mdx +++ b/docs/3.0/deploy/infrastructure-examples/kubernetes.mdx @@ -202,7 +202,7 @@ Switch to the Prefect Cloud UI to create a new Kubernetes work pool. If you set a different namespace, use your selected namespace instead of `prefect` in all commands below. -Note: If there are active workers in your work pool, setup instructions may not be displayed. + You may come back to this page to configure the work pool options at any time. ### Configure work pool options diff --git a/docs/3.0/deploy/infrastructure-examples/serverless.mdx b/docs/3.0/deploy/infrastructure-examples/serverless.mdx index 6f38bc7f498d..2d70b6220bdb 100644 --- a/docs/3.0/deploy/infrastructure-examples/serverless.mdx +++ b/docs/3.0/deploy/infrastructure-examples/serverless.mdx @@ -265,23 +265,90 @@ run this command for your particular cloud provider: to store the client secret value from the generated secret. Here's an abbreviated example output from running the command: -``` + + ```bash + _____________________________________________________________________________________________ + | Provisioning infrastructure for your work pool my-aci-work-pool will require: | + | | + | Updates in subscription Azure subscription 1 | + | | + | - Create a resource group in location eastus | + | - Create an app registration in Azure AD prefect-aci-push-pool-app | + | - Create/use a service principal for app registration | + | - Generate a secret for app registration | + | - Create an Azure Container Registry with prefix prefect | + | - Create an identity prefect-acr-identity to allow access to the created registry | + | - Assign Contributor role to service account | + | - Create an ACR registry for image hosting | + | - Create an identity for Azure Container Instance to allow access to the registry | + | | + | Updates in Prefect workspace | + | | + | - Create Azure Container Instance credentials block aci-push-pool-credentials | + | | + _____________________________________________________________________________________________ + Proceed with infrastructure provisioning? [y/n]: + Creating resource group + Creating app registration + Generating secret for app registration + Creating ACI credentials block + ACI credentials block 'aci-push-pool-credentials' created in Prefect Cloud + Assigning Contributor role to service account + Creating Azure Container Registry + Creating identity + Provisioning infrastructure... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 + Infrastructure successfully provisioned for 'my-aci-work-pool' work pool! + Created work pool 'my-aci-work-pool'! + ``` + + + **Default Docker build namespace** + After infrastructure provisioning completes, you are logged into your new Azure Container Registry and + the default Docker build namespace is set to the URL of the registry. + + + While the default namespace is set, any images you build without specifying a registry or username/organization + are pushed to the registry. + + To use this capability, write your deploy scripts like this: + + ```python example_deploy_script.py + from prefect import flow + from prefect.docker import DockerImage + + + @flow(log_prints=True) + def my_flow(name: str = "world"): + print(f"Hello {name}! I'm a flow running on an Azure Container Instance!") + + + if __name__ == "__main__": + my_flow.deploy( + name="my-deployment", + work_pool_name="my-work-pool", + image=DockerImage( + name="my-image:latest", + platform="linux/amd64", + ) + ) + ``` + + This builds an image with the tag `/my-image:latest` and pushes it to the registry. + + + ```bash -prefect work-pool create --type cloud-run:push --provision-infra my-cloud-run-pool --check-active-workers +prefect work-pool create --type cloud-run:push --provision-infra my-cloud-run-pool ``` The `--provision-infra` flag allows you to select a GCP project to use for your work pool and automatically -configures it to execute flows through Cloud Run. The optional `--check-active-workers` flag enables the system to -detect active workers in the work pool and suppresses unnecessary worker setup instructions if active workers are -already present. +configures it to execute flows through Cloud Run. In your GCP project, this command activates the Cloud Run API, creates a service account, and creates a key for the service account, (if they don't already exist). In your Prefect workspace, this command creates a [`GCPCredentials` block](/integrations/prefect-gcp/index#authenticate-using-a-gcp-credentials-block) to store the service account key. Here's an abbreviated example output from running the command: -``` - ```bash ____________________________________________________________________________________________________________ diff --git a/docs/3.0/resources/upgrade-agents-to-workers.mdx b/docs/3.0/resources/upgrade-agents-to-workers.mdx index 456d245ec718..4ad1d5738595 100644 --- a/docs/3.0/resources/upgrade-agents-to-workers.mdx +++ b/docs/3.0/resources/upgrade-agents-to-workers.mdx @@ -32,7 +32,6 @@ from that pool. - Improved visibility into the status of each worker, including when a worker was started and when it last polled. - Better handling of race conditions for high availability use cases. -- Deployment scripts now intelligently suppress setup instructions when active workers are detected, streamlining the deployment process for environments with existing setups. ### Work pools From e55a7448ed929ff9dead115fcd88f748035c2a90 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" <179508745+promptless[bot]@users.noreply.github.com> Date: Fri, 15 Nov 2024 05:34:01 +0000 Subject: [PATCH 09/13] Docs update (903e1a3) --- .../deploy/infrastructure-examples/docker.mdx | 82 +------------------ .../resources/upgrade-agents-to-workers.mdx | 1 + 2 files changed, 2 insertions(+), 81 deletions(-) diff --git a/docs/3.0/deploy/infrastructure-examples/docker.mdx b/docs/3.0/deploy/infrastructure-examples/docker.mdx index 73fe40da6c04..2951aa6b30c6 100644 --- a/docs/3.0/deploy/infrastructure-examples/docker.mdx +++ b/docs/3.0/deploy/infrastructure-examples/docker.mdx @@ -1,84 +1,4 @@ ---- -title: Run flows in Docker containers -description: Learn how to deploy a flow to a Docker work pool with workers ---- - -In this example, you will set up: - -- a Docker [**work pool**](/3.0/deploy/infrastructure-concepts/work-pools/): stores the infrastructure configuration for your deployment -- a Docker [**worker**](/3.0/deploy/infrastructure-concepts/workers/): process that polls the Prefect API for flow runs to execute as Docker containers -- a [**deployment**](/3.0/deploy/index/): a flow that should run according to the configuration on your Docker work pool - -Then you can execute your deployment via the Prefect API (through the SDK, CLI, UI, etc). - -You must have [Docker](https://docs.docker.com/engine/install/) installed and running on your machine. - - -**Executing flows in a long-lived container** - -This guide shows how to run a flow in an ephemeral container that is removed after the flow run completes. -To instead learn how to run flows in a static, long-lived container, see [this](/3.0/deploy/static-infrastructure-examples/docker/) guide. - - - -### Create a work pool - -A work pool provides default infrastructure configurations that all jobs inherit and can override. -You can adjust many defaults, such as the base Docker image, container cleanup behavior, and resource limits. - -To set up a **Docker** type work pool with the default values, run: - -```bash -prefect work-pool create --type docker my-docker-pool -``` - -... or create the work pool in the UI. - -To confirm the work pool creation was successful, run: - -```bash -prefect work-pool ls -``` - -You should see your new `my-docker-pool` listed in the output. - -Next, check that you can see this work pool in your Prefect UI. -Navigate to the **Work Pools** tab and verify that you see `my-docker-pool` listed. -When you click into `my-docker-pool`, you should see a red status icon signifying that this work pool is not ready. - -To make the work pool ready, you'll need to start a worker. -We'll show how to do this next. - -### Start a worker - -Workers are a lightweight polling process that kick off scheduled flow runs on a specific type of infrastructure (such as Docker). -To start a worker on your local machine, open a new terminal and confirm that your virtual environment has `prefect` installed. - -Run the following command in this new terminal to start the worker: - -```bash -prefect worker start --pool my-docker-pool -``` - -You should see the worker start. -It's now polling the Prefect API to check for any scheduled flow runs it should pick up and then submit for execution. -You'll see your new worker listed in the UI under the **Workers** tab of the Work Pools page with a recent last polled date. -The work pool should have a `Ready` status indicator. - - -**Pro Tip:** - -If `my-docker-pool` does not already exist, the below command will create it for you automatically with the default settings for that work pool type, in this case `docker`. - -```bash -prefect worker start --pool my-docker-pool --type docker -``` - - -Keep this terminal session active for the worker to continue to pick up jobs. -Since you are running this worker locally, the worker will if you close the terminal. -In a production setting this worker should run as a [daemonized or managed process](/3.0/resources/daemonize-processes/). - +None ## Create the deployment From the previous steps, you now have: diff --git a/docs/3.0/resources/upgrade-agents-to-workers.mdx b/docs/3.0/resources/upgrade-agents-to-workers.mdx index 4ad1d5738595..f936577894e8 100644 --- a/docs/3.0/resources/upgrade-agents-to-workers.mdx +++ b/docs/3.0/resources/upgrade-agents-to-workers.mdx @@ -32,6 +32,7 @@ from that pool. - Improved visibility into the status of each worker, including when a worker was started and when it last polled. - Better handling of race conditions for high availability use cases. +- Enhanced deployment logic to check for active workers, suppressing unnecessary worker setup instructions when active workers are present, thus reducing user confusion and streamlining the deployment process. ### Work pools From be8bf54081b4f2edfd1daecfa2f9b6f467660c63 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" <179508745+promptless[bot]@users.noreply.github.com> Date: Fri, 15 Nov 2024 15:43:06 +0000 Subject: [PATCH 10/13] Revert previous docs update --- .../deploy/infrastructure-examples/docker.mdx | 82 ++++++++++++++++++- .../resources/upgrade-agents-to-workers.mdx | 1 - 2 files changed, 81 insertions(+), 2 deletions(-) diff --git a/docs/3.0/deploy/infrastructure-examples/docker.mdx b/docs/3.0/deploy/infrastructure-examples/docker.mdx index 2951aa6b30c6..73fe40da6c04 100644 --- a/docs/3.0/deploy/infrastructure-examples/docker.mdx +++ b/docs/3.0/deploy/infrastructure-examples/docker.mdx @@ -1,4 +1,84 @@ -None +--- +title: Run flows in Docker containers +description: Learn how to deploy a flow to a Docker work pool with workers +--- + +In this example, you will set up: + +- a Docker [**work pool**](/3.0/deploy/infrastructure-concepts/work-pools/): stores the infrastructure configuration for your deployment +- a Docker [**worker**](/3.0/deploy/infrastructure-concepts/workers/): process that polls the Prefect API for flow runs to execute as Docker containers +- a [**deployment**](/3.0/deploy/index/): a flow that should run according to the configuration on your Docker work pool + +Then you can execute your deployment via the Prefect API (through the SDK, CLI, UI, etc). + +You must have [Docker](https://docs.docker.com/engine/install/) installed and running on your machine. + + +**Executing flows in a long-lived container** + +This guide shows how to run a flow in an ephemeral container that is removed after the flow run completes. +To instead learn how to run flows in a static, long-lived container, see [this](/3.0/deploy/static-infrastructure-examples/docker/) guide. + + + +### Create a work pool + +A work pool provides default infrastructure configurations that all jobs inherit and can override. +You can adjust many defaults, such as the base Docker image, container cleanup behavior, and resource limits. + +To set up a **Docker** type work pool with the default values, run: + +```bash +prefect work-pool create --type docker my-docker-pool +``` + +... or create the work pool in the UI. + +To confirm the work pool creation was successful, run: + +```bash +prefect work-pool ls +``` + +You should see your new `my-docker-pool` listed in the output. + +Next, check that you can see this work pool in your Prefect UI. +Navigate to the **Work Pools** tab and verify that you see `my-docker-pool` listed. +When you click into `my-docker-pool`, you should see a red status icon signifying that this work pool is not ready. + +To make the work pool ready, you'll need to start a worker. +We'll show how to do this next. + +### Start a worker + +Workers are a lightweight polling process that kick off scheduled flow runs on a specific type of infrastructure (such as Docker). +To start a worker on your local machine, open a new terminal and confirm that your virtual environment has `prefect` installed. + +Run the following command in this new terminal to start the worker: + +```bash +prefect worker start --pool my-docker-pool +``` + +You should see the worker start. +It's now polling the Prefect API to check for any scheduled flow runs it should pick up and then submit for execution. +You'll see your new worker listed in the UI under the **Workers** tab of the Work Pools page with a recent last polled date. +The work pool should have a `Ready` status indicator. + + +**Pro Tip:** + +If `my-docker-pool` does not already exist, the below command will create it for you automatically with the default settings for that work pool type, in this case `docker`. + +```bash +prefect worker start --pool my-docker-pool --type docker +``` + + +Keep this terminal session active for the worker to continue to pick up jobs. +Since you are running this worker locally, the worker will if you close the terminal. +In a production setting this worker should run as a [daemonized or managed process](/3.0/resources/daemonize-processes/). + ## Create the deployment From the previous steps, you now have: diff --git a/docs/3.0/resources/upgrade-agents-to-workers.mdx b/docs/3.0/resources/upgrade-agents-to-workers.mdx index f936577894e8..4ad1d5738595 100644 --- a/docs/3.0/resources/upgrade-agents-to-workers.mdx +++ b/docs/3.0/resources/upgrade-agents-to-workers.mdx @@ -32,7 +32,6 @@ from that pool. - Improved visibility into the status of each worker, including when a worker was started and when it last polled. - Better handling of race conditions for high availability use cases. -- Enhanced deployment logic to check for active workers, suppressing unnecessary worker setup instructions when active workers are present, thus reducing user confusion and streamlining the deployment process. ### Work pools From e87b1bd2392ea50ebcb10b27e248d986424fef8e Mon Sep 17 00:00:00 2001 From: "promptless[bot]" <179508745+promptless[bot]@users.noreply.github.com> Date: Fri, 15 Nov 2024 15:43:15 +0000 Subject: [PATCH 11/13] Docs update (903e1a3) --- .../3.0/deploy/infrastructure-concepts/store-flow-code.mdx | 7 +++++-- docs/3.0/deploy/infrastructure-concepts/workers.mdx | 1 + docs/3.0/resources/upgrade-agents-to-workers.mdx | 2 ++ 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/3.0/deploy/infrastructure-concepts/store-flow-code.mdx b/docs/3.0/deploy/infrastructure-concepts/store-flow-code.mdx index ce8175beef0a..12cea5143abc 100644 --- a/docs/3.0/deploy/infrastructure-concepts/store-flow-code.mdx +++ b/docs/3.0/deploy/infrastructure-concepts/store-flow-code.mdx @@ -143,6 +143,7 @@ If using the Python `deploy` method with a private repository that references a + ```python bb_no_block.py from prefect import flow @@ -281,6 +282,7 @@ If using the Python `deploy` method with a private repository that references a from prefect.blocks.system import Secret + if __name__ == "__main__": gitlab_repo = GitRepository( url="https://gitlab.com/org/my-private-repo.git", @@ -544,6 +546,7 @@ We also include Python code that shows how to use an existing storage block and To create an `AwsCredentials` block: + 1. Install the [prefect-aws](/integrations/prefect-aws) library with `pip install -U prefect-aws` 1. Register the blocks in prefect-aws with `prefect block register -m prefect_aws` 1. Create a user with a role with read and write permissions to access the bucket. If using the UI, create an access key pair with *IAM -> Users -> Security credentials -> Access keys -> Create access key*. Choose *Use case -> Other* and then copy the *Access key* and *Secret access key* values. @@ -564,7 +567,6 @@ We also include Python code that shows how to use an existing storage block and flow.from_source( source="az://mycontainer/myfolder", entrypoint="my_file.py:my_flow", - ).deploy( name="my-azure-deployment", work_pool_name="my-work-pool", job_variables={"env": {"EXTRA_PIP_PACKAGES": "prefect-azure"} }, @@ -648,6 +650,7 @@ We also include Python code that shows how to use an existing storage block and from prefect import flow + if __name__ == "__main__": flow.from_source( source="gs://my-bucket/my-folder", @@ -810,7 +813,7 @@ The exception is when something the server needs to know about has changed, such Rerun the Python script with `deploy` or run `prefect deploy` from the CLI for YAML-based deployments to update your deployment with the new flow code. -## Flow code storage for deployments created with `serve` +## Flow code storage for deployments created with `serve` The Python `serve` method creates a deployment and a local long-running process to poll for flow runs at the same time. diff --git a/docs/3.0/deploy/infrastructure-concepts/workers.mdx b/docs/3.0/deploy/infrastructure-concepts/workers.mdx index 9ac6379e8698..e6a147c38c15 100644 --- a/docs/3.0/deploy/infrastructure-concepts/workers.mdx +++ b/docs/3.0/deploy/infrastructure-concepts/workers.mdx @@ -143,6 +143,7 @@ For example, to limit a worker to five concurrent flow runs: prefect worker start --pool "my-pool" --limit 5 ``` +If deploying to a work pool with active workers, the worker setup instructions may be omitted. ### Configure prefetch By default, the worker submits flow runs a short time (10 seconds) before they are scheduled to run. diff --git a/docs/3.0/resources/upgrade-agents-to-workers.mdx b/docs/3.0/resources/upgrade-agents-to-workers.mdx index 4ad1d5738595..7d4a2c845b0a 100644 --- a/docs/3.0/resources/upgrade-agents-to-workers.mdx +++ b/docs/3.0/resources/upgrade-agents-to-workers.mdx @@ -42,6 +42,8 @@ execution in your cloud provider environment without the need to host a worker. - Prefect Cloud [managed work pools](/3.0/deploy/infrastructure-examples/managed/) allow you to run flows on Prefect's infrastructure, without the need to host a worker or configure cloud provider infrastructure. +- Deployment process now omits worker setup instructions if deploying to a healthy work-pool with active workers, streamlining user experience. + ### Improved deployment interfaces - The Python deployment experience with `.deploy()` or the alternative deployment experience with From 9d51ce374b6ff93a36078042f1124719a601ce6e Mon Sep 17 00:00:00 2001 From: "promptless[bot]" <179508745+promptless[bot]@users.noreply.github.com> Date: Fri, 15 Nov 2024 16:42:58 +0000 Subject: [PATCH 12/13] Revert previous docs update --- .../3.0/deploy/infrastructure-concepts/store-flow-code.mdx | 7 ++----- docs/3.0/deploy/infrastructure-concepts/workers.mdx | 1 - docs/3.0/resources/upgrade-agents-to-workers.mdx | 2 -- 3 files changed, 2 insertions(+), 8 deletions(-) diff --git a/docs/3.0/deploy/infrastructure-concepts/store-flow-code.mdx b/docs/3.0/deploy/infrastructure-concepts/store-flow-code.mdx index 12cea5143abc..ce8175beef0a 100644 --- a/docs/3.0/deploy/infrastructure-concepts/store-flow-code.mdx +++ b/docs/3.0/deploy/infrastructure-concepts/store-flow-code.mdx @@ -143,7 +143,6 @@ If using the Python `deploy` method with a private repository that references a - ```python bb_no_block.py from prefect import flow @@ -282,7 +281,6 @@ If using the Python `deploy` method with a private repository that references a from prefect.blocks.system import Secret - if __name__ == "__main__": gitlab_repo = GitRepository( url="https://gitlab.com/org/my-private-repo.git", @@ -546,7 +544,6 @@ We also include Python code that shows how to use an existing storage block and To create an `AwsCredentials` block: - 1. Install the [prefect-aws](/integrations/prefect-aws) library with `pip install -U prefect-aws` 1. Register the blocks in prefect-aws with `prefect block register -m prefect_aws` 1. Create a user with a role with read and write permissions to access the bucket. If using the UI, create an access key pair with *IAM -> Users -> Security credentials -> Access keys -> Create access key*. Choose *Use case -> Other* and then copy the *Access key* and *Secret access key* values. @@ -567,6 +564,7 @@ We also include Python code that shows how to use an existing storage block and flow.from_source( source="az://mycontainer/myfolder", entrypoint="my_file.py:my_flow", + ).deploy( name="my-azure-deployment", work_pool_name="my-work-pool", job_variables={"env": {"EXTRA_PIP_PACKAGES": "prefect-azure"} }, @@ -650,7 +648,6 @@ We also include Python code that shows how to use an existing storage block and from prefect import flow - if __name__ == "__main__": flow.from_source( source="gs://my-bucket/my-folder", @@ -813,7 +810,7 @@ The exception is when something the server needs to know about has changed, such Rerun the Python script with `deploy` or run `prefect deploy` from the CLI for YAML-based deployments to update your deployment with the new flow code. -## Flow code storage for deployments created with `serve` +## Flow code storage for deployments created with `serve` The Python `serve` method creates a deployment and a local long-running process to poll for flow runs at the same time. diff --git a/docs/3.0/deploy/infrastructure-concepts/workers.mdx b/docs/3.0/deploy/infrastructure-concepts/workers.mdx index e6a147c38c15..9ac6379e8698 100644 --- a/docs/3.0/deploy/infrastructure-concepts/workers.mdx +++ b/docs/3.0/deploy/infrastructure-concepts/workers.mdx @@ -143,7 +143,6 @@ For example, to limit a worker to five concurrent flow runs: prefect worker start --pool "my-pool" --limit 5 ``` -If deploying to a work pool with active workers, the worker setup instructions may be omitted. ### Configure prefetch By default, the worker submits flow runs a short time (10 seconds) before they are scheduled to run. diff --git a/docs/3.0/resources/upgrade-agents-to-workers.mdx b/docs/3.0/resources/upgrade-agents-to-workers.mdx index 7d4a2c845b0a..4ad1d5738595 100644 --- a/docs/3.0/resources/upgrade-agents-to-workers.mdx +++ b/docs/3.0/resources/upgrade-agents-to-workers.mdx @@ -42,8 +42,6 @@ execution in your cloud provider environment without the need to host a worker. - Prefect Cloud [managed work pools](/3.0/deploy/infrastructure-examples/managed/) allow you to run flows on Prefect's infrastructure, without the need to host a worker or configure cloud provider infrastructure. -- Deployment process now omits worker setup instructions if deploying to a healthy work-pool with active workers, streamlining user experience. - ### Improved deployment interfaces - The Python deployment experience with `.deploy()` or the alternative deployment experience with From 314349ec1392bc277c031bd952712f62557860c0 Mon Sep 17 00:00:00 2001 From: "promptless[bot]" <179508745+promptless[bot]@users.noreply.github.com> Date: Fri, 15 Nov 2024 16:43:13 +0000 Subject: [PATCH 13/13] Docs update (903e1a3) --- docs/3.0/deploy/index.mdx | 4 + .../infrastructure-concepts/work-pools.mdx | 33 +-- .../infrastructure-concepts/workers.mdx | 7 + .../deploy/infrastructure-examples/docker.mdx | 5 +- .../infrastructure-examples/managed.mdx | 13 +- .../infrastructure-examples/serverless.mdx | 230 +++++------------- .../resources/upgrade-agents-to-workers.mdx | 3 + 7 files changed, 80 insertions(+), 215 deletions(-) diff --git a/docs/3.0/deploy/index.mdx b/docs/3.0/deploy/index.mdx index 45bb1b291ac7..2a265fa1b6ac 100644 --- a/docs/3.0/deploy/index.mdx +++ b/docs/3.0/deploy/index.mdx @@ -421,3 +421,7 @@ configuration, and prepares the runtime environment for workflow execution. Pull steps allow users to highly decouple their workflow architecture. For example, a common use of pull steps is to dynamically pull code from remote filesystems such as GitHub with each run of their deployment. + +- **`worker_status_filter`**: (Optional) A filter to determine active workers within the work pool. +This field can be used to prevent unnecessary worker setup instructions during deployment if active workers are present. + \ No newline at end of file diff --git a/docs/3.0/deploy/infrastructure-concepts/work-pools.mdx b/docs/3.0/deploy/infrastructure-concepts/work-pools.mdx index 92f8d43bbff2..5ca2f5cdb25e 100644 --- a/docs/3.0/deploy/infrastructure-concepts/work-pools.mdx +++ b/docs/3.0/deploy/infrastructure-concepts/work-pools.mdx @@ -83,38 +83,6 @@ flowchart LR E -.-> F ``` -### Work pool configuration - -You can configure work pools by using any of the following: - -- Prefect UI -- Prefect CLI commands -- [Prefect REST API](/3.0/api-ref/rest-api/) -- [Terraform provider for Prefect Cloud](https://registry.terraform.io/providers/PrefectHQ/prefect/latest/docs/resources/work_pool) - -To manage work pools in the UI, click the **Work Pools** icon. This displays a list of currently configured work pools. - -![The UI displays a list of configured work pools](/3.0/img/ui/work-pool-list.png) - -Select the **+** button to create a new work pool. You can specify the details about infrastructure created for this work pool. - -To create a work pool through the Prefect CLI, use the `prefect work-pool create` command: - -```bash -prefect work-pool create [OPTIONS] NAME -``` - -`NAME` is a required, unique name for the work pool. - -Optional configuration parameters to filter work on the pool include: - -| Option | Description | -| -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `--paused` | If provided, the work pool is created in a paused state. | -| `--type` | The type of infrastructure that can execute runs from this work pool. | -| `--set-as-default` | Whether to use the created work pool as the local default for deployment. | -| `--base-job-template` | The path to a JSON file containing the base job template to use. If unspecified, Prefect uses the default base job template for the given worker type. | - To update a work pool through the Prefect CLI, use the `prefect work-pool update` command: ```bash @@ -130,6 +98,7 @@ Optional configuration parameters you can specify to update the work pool includ | `--base-job-template` | The path to a JSON file containing the base job template to use. If unspecified, Prefect uses the default base job template for the given worker type. | | `--description` | A description of the work pool. | | `--concurrency-limit` | The maximum number of flow runs to run simultaneously in the work pool. | +| `--worker-filter-status` | Filter workers based on their status using `any_` or `not_any_` fields. | **Managing work pools in CI/CD** diff --git a/docs/3.0/deploy/infrastructure-concepts/workers.mdx b/docs/3.0/deploy/infrastructure-concepts/workers.mdx index 9ac6379e8698..03903dae13bd 100644 --- a/docs/3.0/deploy/infrastructure-concepts/workers.mdx +++ b/docs/3.0/deploy/infrastructure-concepts/workers.mdx @@ -96,6 +96,12 @@ execute a flow run. with the `Worker` role, which you can configure by setting the `PREFECT_API_KEY`. + +**Active Worker Check** + + Before starting a new worker, the system checks for active workers in the specified work pool. If active workers are detected, the system may not suggest starting a new worker, thereby optimizing the deployment process. + + ### Worker status Workers have two statuses: `ONLINE` and `OFFLINE`. A worker is online if it sends regular heartbeat messages to the Prefect API. @@ -143,6 +149,7 @@ For example, to limit a worker to five concurrent flow runs: prefect worker start --pool "my-pool" --limit 5 ``` +If there are already active workers in the work pool, Prefect will not suggest starting new workers. This avoids unnecessary duplication and optimizes resource usage. ### Configure prefetch By default, the worker submits flow runs a short time (10 seconds) before they are scheduled to run. diff --git a/docs/3.0/deploy/infrastructure-examples/docker.mdx b/docs/3.0/deploy/infrastructure-examples/docker.mdx index 73fe40da6c04..f9342f581306 100644 --- a/docs/3.0/deploy/infrastructure-examples/docker.mdx +++ b/docs/3.0/deploy/infrastructure-examples/docker.mdx @@ -60,10 +60,7 @@ Run the following command in this new terminal to start the worker: prefect worker start --pool my-docker-pool ``` -You should see the worker start. -It's now polling the Prefect API to check for any scheduled flow runs it should pick up and then submit for execution. -You'll see your new worker listed in the UI under the **Workers** tab of the Work Pools page with a recent last polled date. -The work pool should have a `Ready` status indicator. +If there are no active workers in the work pool, you should see the worker start. It's now polling the Prefect API to check for any scheduled flow runs it should pick up and then submit for execution. You'll see your new worker listed in the UI under the **Workers** tab of the Work Pools page with a recent last polled date. The work pool should have a `Ready` status indicator. **Pro Tip:** diff --git a/docs/3.0/deploy/infrastructure-examples/managed.mdx b/docs/3.0/deploy/infrastructure-examples/managed.mdx index 479ac1971d95..2b4898a4bc3c 100644 --- a/docs/3.0/deploy/infrastructure-examples/managed.mdx +++ b/docs/3.0/deploy/infrastructure-examples/managed.mdx @@ -1,14 +1,3 @@ ---- -title: Run flows on Prefect Managed infrastructure -description: Learn how Prefect runs deployments on Prefect's infrastructure. ---- - - Prefect Cloud can run your flows on your behalf with Prefect Managed work pools. - -Flows that run with this work pool do not require a worker or cloud provider -account—Prefect handles the infrastructure and code execution for you. - -Managed execution is a great option for users who want to get started quickly, with no infrastructure setup. ## Create a managed deployment @@ -43,7 +32,7 @@ Managed execution is a great option for users who want to get started quickly, w 1. Run the deployment from the UI or from the CLI. - This process runs a flow on remote infrastructure without any infrastructure setup, starting a worker, or requiring a cloud provider account. + This process runs a flow on remote infrastructure without any infrastructure setup, and it will only suggest starting a new worker if no active workers are found in the work pool. ## Add dependencies diff --git a/docs/3.0/deploy/infrastructure-examples/serverless.mdx b/docs/3.0/deploy/infrastructure-examples/serverless.mdx index 2d70b6220bdb..bd7f9bc3117d 100644 --- a/docs/3.0/deploy/infrastructure-examples/serverless.mdx +++ b/docs/3.0/deploy/infrastructure-examples/serverless.mdx @@ -163,192 +163,88 @@ To use automatic infrastructure provisioning, you need: -### Automatically create a new push work pool and provision infrastructure - -To create a new push work pool and configure the necessary infrastructure, -run this command for your particular cloud provider: - - - - - ```bash - prefect work-pool create --type ecs:push --provision-infra my-ecs-pool - ``` - - The `--provision-infra` flag automatically sets up your default AWS account to execute - flows with ECS tasks. - In your AWS account, this command creates a new IAM user, IAM policy, and - ECS cluster that uses AWS Fargate, VPC, and ECR repository (if they don't already exist). - In your Prefect workspace, this command creates an - [`AWSCredentials` block](/integrations/prefect-aws/index#save-credentials-to-an-aws-credentials-block) for storing the generated credentials. - - Here's an abbreviated example output from running the command: - - ```bash - _____________________________________________________________________________________________ - | Provisioning infrastructure for your work pool my-ecs-pool will require: | - | | - | - Creating an IAM user for managing ECS tasks: prefect-ecs-user | - | - Creating and attaching an IAM policy for managing ECS tasks: prefect-ecs-policy | - | - Storing generated AWS credentials in a block | - | - Creating an ECS cluster for running Prefect flows: prefect-ecs-cluster | - | - Creating a VPC with CIDR 172.31.0.0/16 for running ECS tasks: prefect-ecs-vpc | - | - Creating an ECR repository for storing Prefect images: prefect-flows | - _____________________________________________________________________________________________ - Proceed with infrastructure provisioning? [y/n]: y - Provisioning IAM user - Creating IAM policy - Generating AWS credentials - Creating AWS credentials block - Provisioning ECS cluster - Provisioning VPC - Creating internet gateway - Setting up subnets - Setting up security group - Provisioning ECR repository - Authenticating with ECR - Setting default Docker build namespace - Provisioning Infrastructure ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 - Infrastructure successfully provisioned! - Created work pool 'my-ecs-pool'! - ``` - - - **Default Docker build namespace** - - After infrastructure provisioning completes, you will be logged into your new ECR repository and the default - Docker build namespace will be set to the URL of the registry. - - - While the default namespace is set, you do not need to provide the registry URL when building images as - part of your deployment process. - - To take advantage of this, you can write your deploy scripts like this: - - ```python example_deploy_script.py - from prefect import flow - from prefect.docker import DockerImage - - @flow(log_prints=True) - def my_flow(name: str = "world"): - print(f"Hello {name}! I'm a flow running in a ECS task!") - +There are no proposed changes to assess. The document remains unchanged. +``` + +```bash +prefect work-pool create --type azure-container-instances:push --provision-infra my-aci-work-pool +``` - if __name__ == "__main__": - my_flow.deploy( - name="my-deployment", - work_pool_name="my-work-pool", - image=DockerImage( - name="my-repository:latest", - platform="linux/amd64", - ) - ) - ``` +The `--provision-infra` flag enables the automatic setup of necessary infrastructure for deploying flows on Azure Container Instances. This command will now check for active workers in the specified work pool before suggesting any new worker setup instructions. If active workers are found, the deployment process will be streamlined by omitting redundant setup instructions. - This builds an image with the tag `/my-image:latest` and push it to the registry. +In your Azure subscription, this command performs several actions including creating a resource group, app registration, and assigning roles, among others. Additionally, it creates an `AzureContainerInstanceCredentials` block in your Prefect workspace. - Your image name needs to match the name of the repository created with your work pool. You can create - new repositories in the ECR console. - - +Here's an abbreviated example output from running the command: +```bash +Provisioning infrastructure for your work pool my-aci-work-pool will require: - ```bash - prefect work-pool create --type azure-container-instance:push --provision-infra my-aci-pool - ``` + Updates in subscription Azure subscription 1 - The `--provision-infra` flag automatically sets up your default Azure account to execute - flows through Azure Container Instances. - In your Azure account, this command creates a resource group, app registration, service account with necessary permission, - generates a secret for the app registration, and creates an Azure Container Registry, (if they don't already exist). - In your Prefect workspace, this command creates an - [`AzureContainerInstanceCredentials` block](/integrations/prefect-azure) - to store the client secret value from the generated secret. + - Create a resource group in location eastus + - Create an app registration in Azure AD prefect-aci-push-pool-app + - Create/use a service principal for app registration + - Generate a secret for app registration + - Create an Azure Container Registry with prefix prefect + - Create an identity prefect-acr-identity to allow access to the created registry + - Assign Contributor role to service account + - Create an ACR registry for image hosting + - Create an identity for Azure Container Instance to allow access to the registry - Here's an abbreviated example output from running the command: + Updates in Prefect workspace - ```bash - _____________________________________________________________________________________________ - | Provisioning infrastructure for your work pool my-aci-work-pool will require: | - | | - | Updates in subscription Azure subscription 1 | - | | - | - Create a resource group in location eastus | - | - Create an app registration in Azure AD prefect-aci-push-pool-app | - | - Create/use a service principal for app registration | - | - Generate a secret for app registration | - | - Create an Azure Container Registry with prefix prefect | - | - Create an identity prefect-acr-identity to allow access to the created registry | - | - Assign Contributor role to service account | - | - Create an ACR registry for image hosting | - | - Create an identity for Azure Container Instance to allow access to the registry | - | | - | Updates in Prefect workspace | - | | - | - Create Azure Container Instance credentials block aci-push-pool-credentials | - | | - _____________________________________________________________________________________________ - Proceed with infrastructure provisioning? [y/n]: - Creating resource group - Creating app registration - Generating secret for app registration - Creating ACI credentials block - ACI credentials block 'aci-push-pool-credentials' created in Prefect Cloud - Assigning Contributor role to service account - Creating Azure Container Registry - Creating identity - Provisioning infrastructure... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 - Infrastructure successfully provisioned for 'my-aci-work-pool' work pool! - Created work pool 'my-aci-work-pool'! - ``` - - - **Default Docker build namespace** - After infrastructure provisioning completes, you are logged into your new Azure Container Registry and - the default Docker build namespace is set to the URL of the registry. - + - Create Azure Container Instance credentials block aci-push-pool-credentials - While the default namespace is set, any images you build without specifying a registry or username/organization - are pushed to the registry. +Proceed with infrastructure provisioning? [y/n]: +Creating resource group +Creating app registration +Generating secret for app registration +Creating ACI credentials block +ACI credentials block 'aci-push-pool-credentials' created in Prefect Cloud +Assigning Contributor role to service account +Creating Azure Container Registry +Creating identity +Provisioning infrastructure... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 +Infrastructure successfully provisioned for 'my-aci-work-pool' work pool! +Created work pool 'my-aci-work-pool'! +``` - To use this capability, write your deploy scripts like this: + +**Default Docker build namespace** + After infrastructure provisioning completes, you are logged into your new Azure Container Registry and + the default Docker build namespace is set to the URL of the registry. + - ```python example_deploy_script.py - from prefect import flow - from prefect.docker import DockerImage +While the default namespace is set, any images you build without specifying a registry or username/organization +are pushed to the registry. +To use this capability, write your deploy scripts like this: - @flow(log_prints=True) - def my_flow(name: str = "world"): - print(f"Hello {name}! I'm a flow running on an Azure Container Instance!") +```python example_deploy_script.py +from prefect import flow +from prefect.docker import DockerImage - if __name__ == "__main__": - my_flow.deploy( - name="my-deployment", - work_pool_name="my-work-pool", - image=DockerImage( - name="my-image:latest", - platform="linux/amd64", - ) - ) - ``` +@flow(log_prints=True) +def my_flow(name: str = "world"): + print(f"Hello {name}! I'm a flow running on an Azure Container Instance!") - This builds an image with the tag `/my-image:latest` and pushes it to the registry. - - -```bash -prefect work-pool create --type cloud-run:push --provision-infra my-cloud-run-pool +if __name__ == "__main__": + my_flow.deploy( + name="my-deployment", + work_pool_name="my-work-pool", + image=DockerImage( + name="my-image:latest", + platform="linux/amd64", + ) + ) ``` -The `--provision-infra` flag allows you to select a GCP project to use for your work pool and automatically -configures it to execute flows through Cloud Run. -In your GCP project, this command activates the Cloud Run API, creates a service account, and creates a key for the -service account, (if they don't already exist). -In your Prefect workspace, this command creates a -[`GCPCredentials` block](/integrations/prefect-gcp/index#authenticate-using-a-gcp-credentials-block) to store the service account key. +This builds an image with the tag `/my-image:latest` and pushes it to the registry. +``` + +``` -Here's an abbreviated example output from running the command: ```bash ____________________________________________________________________________________________________________ diff --git a/docs/3.0/resources/upgrade-agents-to-workers.mdx b/docs/3.0/resources/upgrade-agents-to-workers.mdx index 4ad1d5738595..17b3ab7cb2aa 100644 --- a/docs/3.0/resources/upgrade-agents-to-workers.mdx +++ b/docs/3.0/resources/upgrade-agents-to-workers.mdx @@ -32,6 +32,7 @@ from that pool. - Improved visibility into the status of each worker, including when a worker was started and when it last polled. - Better handling of race conditions for high availability use cases. +- Enhanced deployment process with an active worker check, reducing unnecessary setup instructions for users with existing active workers. ### Work pools @@ -155,6 +156,8 @@ by workers. This worker replaces your agent and polls your new work pool for flow runs to execute. +If there are already active workers in your work pool, you will not need to start a new worker. + ```bash prefect worker start -p ```