diff --git a/docs/3.0/deploy/index.mdx b/docs/3.0/deploy/index.mdx index 45bb1b291ac7..2a265fa1b6ac 100644 --- a/docs/3.0/deploy/index.mdx +++ b/docs/3.0/deploy/index.mdx @@ -421,3 +421,7 @@ configuration, and prepares the runtime environment for workflow execution. Pull steps allow users to highly decouple their workflow architecture. For example, a common use of pull steps is to dynamically pull code from remote filesystems such as GitHub with each run of their deployment. + +- **`worker_status_filter`**: (Optional) A filter to determine active workers within the work pool. +This field can be used to prevent unnecessary worker setup instructions during deployment if active workers are present. + \ No newline at end of file diff --git a/docs/3.0/deploy/infrastructure-concepts/work-pools.mdx b/docs/3.0/deploy/infrastructure-concepts/work-pools.mdx index 92f8d43bbff2..5ca2f5cdb25e 100644 --- a/docs/3.0/deploy/infrastructure-concepts/work-pools.mdx +++ b/docs/3.0/deploy/infrastructure-concepts/work-pools.mdx @@ -83,38 +83,6 @@ flowchart LR E -.-> F ``` -### Work pool configuration - -You can configure work pools by using any of the following: - -- Prefect UI -- Prefect CLI commands -- [Prefect REST API](/3.0/api-ref/rest-api/) -- [Terraform provider for Prefect Cloud](https://registry.terraform.io/providers/PrefectHQ/prefect/latest/docs/resources/work_pool) - -To manage work pools in the UI, click the **Work Pools** icon. This displays a list of currently configured work pools. - -![The UI displays a list of configured work pools](/3.0/img/ui/work-pool-list.png) - -Select the **+** button to create a new work pool. You can specify the details about infrastructure created for this work pool. - -To create a work pool through the Prefect CLI, use the `prefect work-pool create` command: - -```bash -prefect work-pool create [OPTIONS] NAME -``` - -`NAME` is a required, unique name for the work pool. - -Optional configuration parameters to filter work on the pool include: - -| Option | Description | -| -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `--paused` | If provided, the work pool is created in a paused state. | -| `--type` | The type of infrastructure that can execute runs from this work pool. | -| `--set-as-default` | Whether to use the created work pool as the local default for deployment. | -| `--base-job-template` | The path to a JSON file containing the base job template to use. If unspecified, Prefect uses the default base job template for the given worker type. | - To update a work pool through the Prefect CLI, use the `prefect work-pool update` command: ```bash @@ -130,6 +98,7 @@ Optional configuration parameters you can specify to update the work pool includ | `--base-job-template` | The path to a JSON file containing the base job template to use. If unspecified, Prefect uses the default base job template for the given worker type. | | `--description` | A description of the work pool. | | `--concurrency-limit` | The maximum number of flow runs to run simultaneously in the work pool. | +| `--worker-filter-status` | Filter workers based on their status using `any_` or `not_any_` fields. | **Managing work pools in CI/CD** diff --git a/docs/3.0/deploy/infrastructure-concepts/workers.mdx b/docs/3.0/deploy/infrastructure-concepts/workers.mdx index 9ac6379e8698..03903dae13bd 100644 --- a/docs/3.0/deploy/infrastructure-concepts/workers.mdx +++ b/docs/3.0/deploy/infrastructure-concepts/workers.mdx @@ -96,6 +96,12 @@ execute a flow run. with the `Worker` role, which you can configure by setting the `PREFECT_API_KEY`. + +**Active Worker Check** + + Before starting a new worker, the system checks for active workers in the specified work pool. If active workers are detected, the system may not suggest starting a new worker, thereby optimizing the deployment process. + + ### Worker status Workers have two statuses: `ONLINE` and `OFFLINE`. A worker is online if it sends regular heartbeat messages to the Prefect API. @@ -143,6 +149,7 @@ For example, to limit a worker to five concurrent flow runs: prefect worker start --pool "my-pool" --limit 5 ``` +If there are already active workers in the work pool, Prefect will not suggest starting new workers. This avoids unnecessary duplication and optimizes resource usage. ### Configure prefetch By default, the worker submits flow runs a short time (10 seconds) before they are scheduled to run. diff --git a/docs/3.0/deploy/infrastructure-examples/docker.mdx b/docs/3.0/deploy/infrastructure-examples/docker.mdx index 73fe40da6c04..f9342f581306 100644 --- a/docs/3.0/deploy/infrastructure-examples/docker.mdx +++ b/docs/3.0/deploy/infrastructure-examples/docker.mdx @@ -60,10 +60,7 @@ Run the following command in this new terminal to start the worker: prefect worker start --pool my-docker-pool ``` -You should see the worker start. -It's now polling the Prefect API to check for any scheduled flow runs it should pick up and then submit for execution. -You'll see your new worker listed in the UI under the **Workers** tab of the Work Pools page with a recent last polled date. -The work pool should have a `Ready` status indicator. +If there are no active workers in the work pool, you should see the worker start. It's now polling the Prefect API to check for any scheduled flow runs it should pick up and then submit for execution. You'll see your new worker listed in the UI under the **Workers** tab of the Work Pools page with a recent last polled date. The work pool should have a `Ready` status indicator. **Pro Tip:** diff --git a/docs/3.0/deploy/infrastructure-examples/managed.mdx b/docs/3.0/deploy/infrastructure-examples/managed.mdx index 479ac1971d95..2b4898a4bc3c 100644 --- a/docs/3.0/deploy/infrastructure-examples/managed.mdx +++ b/docs/3.0/deploy/infrastructure-examples/managed.mdx @@ -1,14 +1,3 @@ ---- -title: Run flows on Prefect Managed infrastructure -description: Learn how Prefect runs deployments on Prefect's infrastructure. ---- - - Prefect Cloud can run your flows on your behalf with Prefect Managed work pools. - -Flows that run with this work pool do not require a worker or cloud provider -account—Prefect handles the infrastructure and code execution for you. - -Managed execution is a great option for users who want to get started quickly, with no infrastructure setup. ## Create a managed deployment @@ -43,7 +32,7 @@ Managed execution is a great option for users who want to get started quickly, w 1. Run the deployment from the UI or from the CLI. - This process runs a flow on remote infrastructure without any infrastructure setup, starting a worker, or requiring a cloud provider account. + This process runs a flow on remote infrastructure without any infrastructure setup, and it will only suggest starting a new worker if no active workers are found in the work pool. ## Add dependencies diff --git a/docs/3.0/deploy/infrastructure-examples/serverless.mdx b/docs/3.0/deploy/infrastructure-examples/serverless.mdx index 2d70b6220bdb..bd7f9bc3117d 100644 --- a/docs/3.0/deploy/infrastructure-examples/serverless.mdx +++ b/docs/3.0/deploy/infrastructure-examples/serverless.mdx @@ -163,192 +163,88 @@ To use automatic infrastructure provisioning, you need: -### Automatically create a new push work pool and provision infrastructure - -To create a new push work pool and configure the necessary infrastructure, -run this command for your particular cloud provider: - - - - - ```bash - prefect work-pool create --type ecs:push --provision-infra my-ecs-pool - ``` - - The `--provision-infra` flag automatically sets up your default AWS account to execute - flows with ECS tasks. - In your AWS account, this command creates a new IAM user, IAM policy, and - ECS cluster that uses AWS Fargate, VPC, and ECR repository (if they don't already exist). - In your Prefect workspace, this command creates an - [`AWSCredentials` block](/integrations/prefect-aws/index#save-credentials-to-an-aws-credentials-block) for storing the generated credentials. - - Here's an abbreviated example output from running the command: - - ```bash - _____________________________________________________________________________________________ - | Provisioning infrastructure for your work pool my-ecs-pool will require: | - | | - | - Creating an IAM user for managing ECS tasks: prefect-ecs-user | - | - Creating and attaching an IAM policy for managing ECS tasks: prefect-ecs-policy | - | - Storing generated AWS credentials in a block | - | - Creating an ECS cluster for running Prefect flows: prefect-ecs-cluster | - | - Creating a VPC with CIDR 172.31.0.0/16 for running ECS tasks: prefect-ecs-vpc | - | - Creating an ECR repository for storing Prefect images: prefect-flows | - _____________________________________________________________________________________________ - Proceed with infrastructure provisioning? [y/n]: y - Provisioning IAM user - Creating IAM policy - Generating AWS credentials - Creating AWS credentials block - Provisioning ECS cluster - Provisioning VPC - Creating internet gateway - Setting up subnets - Setting up security group - Provisioning ECR repository - Authenticating with ECR - Setting default Docker build namespace - Provisioning Infrastructure ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 - Infrastructure successfully provisioned! - Created work pool 'my-ecs-pool'! - ``` - - - **Default Docker build namespace** - - After infrastructure provisioning completes, you will be logged into your new ECR repository and the default - Docker build namespace will be set to the URL of the registry. - - - While the default namespace is set, you do not need to provide the registry URL when building images as - part of your deployment process. - - To take advantage of this, you can write your deploy scripts like this: - - ```python example_deploy_script.py - from prefect import flow - from prefect.docker import DockerImage - - @flow(log_prints=True) - def my_flow(name: str = "world"): - print(f"Hello {name}! I'm a flow running in a ECS task!") - +There are no proposed changes to assess. The document remains unchanged. +``` + +```bash +prefect work-pool create --type azure-container-instances:push --provision-infra my-aci-work-pool +``` - if __name__ == "__main__": - my_flow.deploy( - name="my-deployment", - work_pool_name="my-work-pool", - image=DockerImage( - name="my-repository:latest", - platform="linux/amd64", - ) - ) - ``` +The `--provision-infra` flag enables the automatic setup of necessary infrastructure for deploying flows on Azure Container Instances. This command will now check for active workers in the specified work pool before suggesting any new worker setup instructions. If active workers are found, the deployment process will be streamlined by omitting redundant setup instructions. - This builds an image with the tag `/my-image:latest` and push it to the registry. +In your Azure subscription, this command performs several actions including creating a resource group, app registration, and assigning roles, among others. Additionally, it creates an `AzureContainerInstanceCredentials` block in your Prefect workspace. - Your image name needs to match the name of the repository created with your work pool. You can create - new repositories in the ECR console. - - +Here's an abbreviated example output from running the command: +```bash +Provisioning infrastructure for your work pool my-aci-work-pool will require: - ```bash - prefect work-pool create --type azure-container-instance:push --provision-infra my-aci-pool - ``` + Updates in subscription Azure subscription 1 - The `--provision-infra` flag automatically sets up your default Azure account to execute - flows through Azure Container Instances. - In your Azure account, this command creates a resource group, app registration, service account with necessary permission, - generates a secret for the app registration, and creates an Azure Container Registry, (if they don't already exist). - In your Prefect workspace, this command creates an - [`AzureContainerInstanceCredentials` block](/integrations/prefect-azure) - to store the client secret value from the generated secret. + - Create a resource group in location eastus + - Create an app registration in Azure AD prefect-aci-push-pool-app + - Create/use a service principal for app registration + - Generate a secret for app registration + - Create an Azure Container Registry with prefix prefect + - Create an identity prefect-acr-identity to allow access to the created registry + - Assign Contributor role to service account + - Create an ACR registry for image hosting + - Create an identity for Azure Container Instance to allow access to the registry - Here's an abbreviated example output from running the command: + Updates in Prefect workspace - ```bash - _____________________________________________________________________________________________ - | Provisioning infrastructure for your work pool my-aci-work-pool will require: | - | | - | Updates in subscription Azure subscription 1 | - | | - | - Create a resource group in location eastus | - | - Create an app registration in Azure AD prefect-aci-push-pool-app | - | - Create/use a service principal for app registration | - | - Generate a secret for app registration | - | - Create an Azure Container Registry with prefix prefect | - | - Create an identity prefect-acr-identity to allow access to the created registry | - | - Assign Contributor role to service account | - | - Create an ACR registry for image hosting | - | - Create an identity for Azure Container Instance to allow access to the registry | - | | - | Updates in Prefect workspace | - | | - | - Create Azure Container Instance credentials block aci-push-pool-credentials | - | | - _____________________________________________________________________________________________ - Proceed with infrastructure provisioning? [y/n]: - Creating resource group - Creating app registration - Generating secret for app registration - Creating ACI credentials block - ACI credentials block 'aci-push-pool-credentials' created in Prefect Cloud - Assigning Contributor role to service account - Creating Azure Container Registry - Creating identity - Provisioning infrastructure... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 - Infrastructure successfully provisioned for 'my-aci-work-pool' work pool! - Created work pool 'my-aci-work-pool'! - ``` - - - **Default Docker build namespace** - After infrastructure provisioning completes, you are logged into your new Azure Container Registry and - the default Docker build namespace is set to the URL of the registry. - + - Create Azure Container Instance credentials block aci-push-pool-credentials - While the default namespace is set, any images you build without specifying a registry or username/organization - are pushed to the registry. +Proceed with infrastructure provisioning? [y/n]: +Creating resource group +Creating app registration +Generating secret for app registration +Creating ACI credentials block +ACI credentials block 'aci-push-pool-credentials' created in Prefect Cloud +Assigning Contributor role to service account +Creating Azure Container Registry +Creating identity +Provisioning infrastructure... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 +Infrastructure successfully provisioned for 'my-aci-work-pool' work pool! +Created work pool 'my-aci-work-pool'! +``` - To use this capability, write your deploy scripts like this: + +**Default Docker build namespace** + After infrastructure provisioning completes, you are logged into your new Azure Container Registry and + the default Docker build namespace is set to the URL of the registry. + - ```python example_deploy_script.py - from prefect import flow - from prefect.docker import DockerImage +While the default namespace is set, any images you build without specifying a registry or username/organization +are pushed to the registry. +To use this capability, write your deploy scripts like this: - @flow(log_prints=True) - def my_flow(name: str = "world"): - print(f"Hello {name}! I'm a flow running on an Azure Container Instance!") +```python example_deploy_script.py +from prefect import flow +from prefect.docker import DockerImage - if __name__ == "__main__": - my_flow.deploy( - name="my-deployment", - work_pool_name="my-work-pool", - image=DockerImage( - name="my-image:latest", - platform="linux/amd64", - ) - ) - ``` +@flow(log_prints=True) +def my_flow(name: str = "world"): + print(f"Hello {name}! I'm a flow running on an Azure Container Instance!") - This builds an image with the tag `/my-image:latest` and pushes it to the registry. - - -```bash -prefect work-pool create --type cloud-run:push --provision-infra my-cloud-run-pool +if __name__ == "__main__": + my_flow.deploy( + name="my-deployment", + work_pool_name="my-work-pool", + image=DockerImage( + name="my-image:latest", + platform="linux/amd64", + ) + ) ``` -The `--provision-infra` flag allows you to select a GCP project to use for your work pool and automatically -configures it to execute flows through Cloud Run. -In your GCP project, this command activates the Cloud Run API, creates a service account, and creates a key for the -service account, (if they don't already exist). -In your Prefect workspace, this command creates a -[`GCPCredentials` block](/integrations/prefect-gcp/index#authenticate-using-a-gcp-credentials-block) to store the service account key. +This builds an image with the tag `/my-image:latest` and pushes it to the registry. +``` + +``` -Here's an abbreviated example output from running the command: ```bash ____________________________________________________________________________________________________________ diff --git a/docs/3.0/resources/upgrade-agents-to-workers.mdx b/docs/3.0/resources/upgrade-agents-to-workers.mdx index 4ad1d5738595..17b3ab7cb2aa 100644 --- a/docs/3.0/resources/upgrade-agents-to-workers.mdx +++ b/docs/3.0/resources/upgrade-agents-to-workers.mdx @@ -32,6 +32,7 @@ from that pool. - Improved visibility into the status of each worker, including when a worker was started and when it last polled. - Better handling of race conditions for high availability use cases. +- Enhanced deployment process with an active worker check, reducing unnecessary setup instructions for users with existing active workers. ### Work pools @@ -155,6 +156,8 @@ by workers. This worker replaces your agent and polls your new work pool for flow runs to execute. +If there are already active workers in your work pool, you will not need to start a new worker. + ```bash prefect worker start -p ```