This repository contains common and extensible Prefect patterns to drive efficient workflows — we like to call these patterns our recipes
Here you'll find starter code and more advanced example use cases.
We're always looking for new contributions! See our existing Recipe Ideas/Issues for inspiration. Read a detailed guide on how to share your solutions with the Prefect community or run these commands to get started right away.
To report issues, typos, or link fixes, please open an issue. We appreciate it!
What are you interested in seeing examples of? Jot down your big idea here.
- Run Multiple Subflows or Child Flows in Parallel
- Subflow with a Different Task Runner Than Parent Flow
- Create a Subflow and Block Until It's Completed
- Running Subflows On Their Own Infrastructure Using a Separate Deployment
- Conditionally Stop a Task Run
- Ensure Tasks Immediately Fail If Upstream Task Fails
- Define State Dependencies Between Tasks
- Serverless Real-Time Data Pipelines on AWS with Prefect, ECS and GitHub Actions
- Build a Data Platform with Prefect, dbt, and Snowflake (using blocks)
- Real World Python for Data Engineering - Supercharge Your Data Orchestration with Prefect 2.0
- Create a Maintainable Data Pipeline with Prefect and DVC
- Data engineering & orchestration with Prefect, Docker, Terraform, Google CloudRun, BigQuery and Streamlit
- Orchestrating Airbyte with Prefect 2
- Coordinate ELT in 2023 with Airbyte, dbt and Prefect
- Schedule & orchestrate dbt Cloud jobs with Prefect
- Prefect & Fivetran: integrate all the tools & orchestrate them in Python
- Export Airbyte Configuration and Load to S3 bucket using blocks, including Python-based deployment
- ELT with Snowflake Using Async and Blocks
- Deploy a Prefect agent to ECS using the AWS CLI
- Deploy a Prefect agent to ECS with Terraform
- Deploy Flows Using ECSTask Infrastructure Blocks
- Deploy a Prefect agent to ECS Fargate using CloudFormation and GitHub Actions
- Setup Azure with Prefect
- Deploy Prefect Orion to an AKS Cluster with Azure Blob Storage
- Setup an Azure VM and Run the Prefect Agent
- Deploy Flows Using Azure Container Instances Infrastructure Blocks
- Conditionally deploy Prefect flow(s) only when flow-related files have changed
- Conditionally build a docker image & deploy Prefect flow(s) only when flow-related files have changed
- Deploy Prefect flows as containers stored in AWS ECR
- Create Observable and Reproducible Notebooks with Hex - Article
- Create Observable and Reproducible Notebooks with Hex - Video
- How to Structure an ML Project for Reproducibility and Maintainability
- Orchestrate Your Data Science Project with Prefect 2.0
- Build a Full-Stack ML Application With Pydantic And Prefect
- How to Build a Modular Data Stack — Data Platform with Prefect, dbt and Snowflake
- How to Build Modular Dataflows with Tasks, Flows and Subflows in Prefect
We're always looking for new contributions! You can add your Prefect 2.0 recipe and earn some swag in a few simple steps:
- Clone the prefect-recipes repo:
git clone [email protected]:PrefectHQ/prefect-recipes.git- Create and checkout a new branch:
git checkout -b feat/new-recipe-name- Add your code under the appropriate category. Unsure? Add it under
flows-advanced/. - Add your recipe to this README.
- Commit and push the code to your remote branch.
- Create a PR 🤌
We use our Slack Community and Discourse to discuss all things Prefect-- such as FAQ, use cases and integrations. Join in the conversation 😄
Thank you for your contributions and efforts to improve prefect-recipes. We're glad to have you in our community!