Skip to content

Commit

Permalink
Update unreleased documentation (#626)
Browse files Browse the repository at this point in the history
* Update versions.json

* Deployed 9dec435 to unreleased in versions with MkDocs 1.5.3 and mike 2.0.0

* Sort docs versions

---------

Co-authored-by: GitHub Actions Bot <[email protected]>
  • Loading branch information
github-actions[bot] and GitHub Actions Bot authored Feb 16, 2024
1 parent 78332ec commit 52f7910
Show file tree
Hide file tree
Showing 14 changed files with 294 additions and 444 deletions.
2 changes: 1 addition & 1 deletion versions/unreleased/guides/big-data/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -9047,7 +9047,7 @@



<h1 id="big-data-with-prefect">Big data with Prefect<a class="headerlink" href="#big-data-with-prefect" title="Permanent link">&para;</a></h1>
<h1 id="big-data-with-prefect">Big Data with Prefect<a class="headerlink" href="#big-data-with-prefect" title="Permanent link">&para;</a></h1>
<p>In this guide you'll learn tips for working with large amounts of data in Prefect.</p>
<p>Big data doesn't have a widely accepted, precise definition.
In this guide, we'll discuss methods to reduce the processing time or memory utilization of Prefect workflows, without editing your Python code.</p>
Expand Down
2 changes: 1 addition & 1 deletion versions/unreleased/guides/ci-cd/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -9047,7 +9047,7 @@



<h1 id="cicd-with-prefect">CI/CD with Prefect<a class="headerlink" href="#cicd-with-prefect" title="Permanent link">&para;</a></h1>
<h1 id="cicd-with-prefect">CI/CD With Prefect<a class="headerlink" href="#cicd-with-prefect" title="Permanent link">&para;</a></h1>
<p>Many organizations deploy Prefect workflows via their CI/CD process.
Each organization has their own unique CI/CD setup, but a common pattern is to use CI/CD to manage Prefect <a href="/concepts/deployments">deployments</a>.
Combining Prefect's deployment features with CI/CD tools enables efficient management of flow code updates, scheduling changes, and container builds.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8906,7 +8906,7 @@


<h1 id="run-deployments-on-serverless-infrastructure-with-prefect-workers">Run Deployments on Serverless Infrastructure with Prefect Workers<a class="headerlink" href="#run-deployments-on-serverless-infrastructure-with-prefect-workers" title="Permanent link">&para;</a></h1>
<p>Prefect provides work pools for workers to run flows on the serverless platforms of major cloud providers.
<p>Prefect provides hybrid work pools for workers to run flows on the serverless platforms of major cloud providers.
The following options are available:</p>
<ul>
<li>AWS Elastic Container Service (ECS)</li>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9137,17 +9137,17 @@



<h1 id="global-concurrency-limits-and-rate-limits">Global concurrency limits and rate limits<a class="headerlink" href="#global-concurrency-limits-and-rate-limits" title="Permanent link">&para;</a></h1>
<h1 id="global-concurrency-limits-and-rate-limits">Global Concurrency Limits and Rate Limits<a class="headerlink" href="#global-concurrency-limits-and-rate-limits" title="Permanent link">&para;</a></h1>
<p>Global concurrency limits allow you to manage execution efficiently, controlling how many tasks, flows, or other operations can run simultaneously. They are ideal when optimizing resource usage, preventing bottlenecks, and customizing task execution are priorities.</p>
<div class="admonition tip">
<p class="admonition-title">Clarification on use of the term 'tasks'</p>
<p>In the context of global concurrency and rate limits, "tasks" refers not specifically to Prefect tasks, but to concurrent units of work in general, such as those managed by an event loop or <code>TaskGroup</code> in asynchronous programming. These general "tasks" could include Prefect tasks when they are part of an asynchronous execution environment.</p>
</div>
<p>Rate Limits ensure system stability by governing the frequency of requests or operations. They are suitable for preventing overuse, ensuring fairness, and handling errors gracefully.</p>
<p>When selecting between Concurrency and Rate Limits, consider your primary goal. Choose Concurrency Limits for resource optimization and task management. Choose Rate Limits to maintain system stability and fair access to services.</p>
<p>The core difference between a rate limit and a concurrency limit is the way in which slots are released. With a rate limit, slots are released at a controlled rate, controlled by <code>slot_decay_per_second</code> whereas with a concurrency limit, slots are released when the concurrency manager is exited. </p>
<p>The core difference between a rate limit and a concurrency limit is the way in which slots are released. With a rate limit, slots are released at a controlled rate, controlled by <code>slot_decay_per_second</code> whereas with a concurrency limit, slots are released when the concurrency manager is exited.</p>
<h2 id="managing-global-concurrency-limits-and-rate-limits">Managing Global concurrency limits and rate limits<a class="headerlink" href="#managing-global-concurrency-limits-and-rate-limits" title="Permanent link">&para;</a></h2>
<p>You can create, read, edit, and delete concurrency limits via the Prefect UI. </p>
<p>You can create, read, edit, and delete concurrency limits via the Prefect UI.</p>
<p>When creating a concurrency limit, you can specify the following parameters:</p>
<ul>
<li><strong>Name</strong>: The name of the concurrency limit. This name is also how you'll reference the concurrency limit in your code. Special characters, such as <code>/</code>, <code>%</code>, <code>&amp;</code>, <code>&gt;</code>, <code>&lt;</code>, are not allowed.</li>
Expand Down Expand Up @@ -9350,7 +9350,7 @@ <h3 id="managing-database-connections">Managing database connections<a class="he
</code></pre></div>
<h3 id="parallel-data-processing">Parallel data processing<a class="headerlink" href="#parallel-data-processing" title="Permanent link">&para;</a></h3>
<p>Limiting the maximum number of parallel processing tasks.</p>
<p>In this scenario we want to limit the number of <code>process_data</code> tasks to five at any one time. We do this by using the <code>concurrency</code> context manager to request five slots on the <code>data-processing</code> concurrency limit. This will block until five slots are free and then submit five more tasks, ensuring that we never exceed the maximum number of parallel processing tasks. </p>
<p>In this scenario we want to limit the number of <code>process_data</code> tasks to five at any one time. We do this by using the <code>concurrency</code> context manager to request five slots on the <code>data-processing</code> concurrency limit. This will block until five slots are free and then submit five more tasks, ensuring that we never exceed the maximum number of parallel processing tasks.</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">asyncio</span>
<span class="kn">from</span> <span class="nn">prefect.concurrency.sync</span> <span class="kn">import</span> <span class="n">concurrency</span>

Expand Down
22 changes: 11 additions & 11 deletions versions/unreleased/guides/host/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,7 @@
<div data-md-component="skip">


<a href="#hosting-a-prefect-server" class="md-skip">
<a href="#hosting-a-prefect-server-instance" class="md-skip">
Skip to content
</a>

Expand Down Expand Up @@ -965,11 +965,11 @@
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>

<li class="md-nav__item">
<a href="#differences-between-a-self-hosted-prefect-server-and-prefect-cloud" class="md-nav__link">
<a href="#differences-between-a-self-hosted-prefect-server-instance-and-prefect-cloud" class="md-nav__link">
<span class="md-ellipsis">

<span class="md-typeset">
Differences between a self-hosted Prefect server and Prefect Cloud
Differences between a self-hosted Prefect server instance and Prefect Cloud
</span>

</span>
Expand Down Expand Up @@ -8937,11 +8937,11 @@
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>

<li class="md-nav__item">
<a href="#differences-between-a-self-hosted-prefect-server-and-prefect-cloud" class="md-nav__link">
<a href="#differences-between-a-self-hosted-prefect-server-instance-and-prefect-cloud" class="md-nav__link">
<span class="md-ellipsis">

<span class="md-typeset">
Differences between a self-hosted Prefect server and Prefect Cloud
Differences between a self-hosted Prefect server instance and Prefect Cloud
</span>

</span>
Expand Down Expand Up @@ -9163,11 +9163,11 @@



<h1 id="hosting-a-prefect-server">Hosting a Prefect server<a class="headerlink" href="#hosting-a-prefect-server" title="Permanent link">&para;</a></h1>
<p>After you install Prefect you have a Python SDK client that can communicate with <a href="https://app.prefect.cloud">Prefect Cloud</a>, the platform hosted by Prefect. You also have an <a href="/api-ref/">API server</a> backed by a database and a UI.</p>
<p>In this section you'll learn how to host your own Prefect server.
<h1 id="hosting-a-prefect-server-instance">Hosting a Prefect server instance<a class="headerlink" href="#hosting-a-prefect-server-instance" title="Permanent link">&para;</a></h1>
<p>After you install Prefect you have a Python SDK client that can communicate with <a href="https://app.prefect.cloud">Prefect Cloud</a>, the platform hosted by Prefect. You also have an <a href="/api-ref/">API server</a> instance backed by a database and a UI.</p>
<p>In this section you'll learn how to host your own Prefect server instance.
If you would like to host a Prefect server instance on Kubernetes, check out the prefect-server <a href="https://github.com/PrefectHQ/prefect-helm/tree/main/charts/prefect-server">Helm chart</a>.</p>
<p>Spin up a local Prefect server UI with the <code>prefect server start</code> CLI command in the terminal:</p>
<p>Spin up a local Prefect server UI by running the <code>prefect server start</code> CLI command in the terminal:</p>
<div class="terminal">
<div class="highlight"><pre><span></span><code>prefect<span class="w"> </span>server<span class="w"> </span>start
</code></pre></div>
Expand All @@ -9176,8 +9176,8 @@ <h1 id="hosting-a-prefect-server">Hosting a Prefect server<a class="headerlink"
<p>Open the URL for the Prefect server UI (<a href="http://127.0.0.1:4200">http://127.0.0.1:4200</a> by default) in a browser.</p>
<p><img alt="Viewing the dashboard in the Prefect UI." src="/img/ui/self-hosted-server-dashboard.png" /></p>
<p>Shut down the Prefect server with <kdb> ctrl </kbd> + <kdb> c </kbd> in the terminal.</p>
<h3 id="differences-between-a-self-hosted-prefect-server-and-prefect-cloud">Differences between a self-hosted Prefect server and Prefect Cloud<a class="headerlink" href="#differences-between-a-self-hosted-prefect-server-and-prefect-cloud" title="Permanent link">&para;</a></h3>
<p>A self-hosted Prefect server and Prefect Cloud share a common set of features. Prefect Cloud includes the following additional features:</p>
<h3 id="differences-between-a-self-hosted-prefect-server-instance-and-prefect-cloud">Differences between a self-hosted Prefect server instance and Prefect Cloud<a class="headerlink" href="#differences-between-a-self-hosted-prefect-server-instance-and-prefect-cloud" title="Permanent link">&para;</a></h3>
<p>A self-hosted Prefect server instance and Prefect Cloud share a common set of features. Prefect Cloud includes the following additional features:</p>
<ul>
<li><a href="/cloud/workspaces/">Workspaces</a> &mdash; isolated environments to organize your flows, deployments, and flow runs.</li>
<li><a href="/cloud/automations/">Automations</a> &mdash; configure triggers, actions, and notifications in response to real-time monitoring events.</li>
Expand Down
23 changes: 15 additions & 8 deletions versions/unreleased/guides/managed-execution/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -9190,7 +9190,9 @@


<h1 id="managed-execution">Managed Execution <span class="badge cloud"></span> <span class="badge beta"/><a class="headerlink" href="#managed-execution" title="Permanent link">&para;</a></h1>
<p>Prefect Cloud can run your flows on your behalf with <strong>prefect:managed</strong> work pools. Flows run with this work pool do not require a worker or cloud provider account. Prefect handles the infrastructure and code execution for you.</p>
<p>Prefect Cloud can run your flows on your behalf with Prefect Managed work pools.
Flows run with this work pool do not require a worker or cloud provider account.
Prefect handles the infrastructure and code execution for you.</p>
<p>Managed execution is a great option for users who want to get started quickly, with no infrastructure setup.</p>
<div class="admonition warning">
<p class="admonition-title">Managed Execution is in beta</p>
Expand All @@ -9200,9 +9202,13 @@ <h1 id="managed-execution">Managed Execution <span class="badge cloud"></span> <
<h2 id="usage-guide">Usage guide<a class="headerlink" href="#usage-guide" title="Permanent link">&para;</a></h2>
<p>Run a flow with managed infrastructure in three steps.</p>
<h3 id="step-1">Step 1<a class="headerlink" href="#step-1" title="Permanent link">&para;</a></h3>
<p>Create a new work pool of type <strong>prefect:managed</strong>. you can do this via the UI wizard, or via the CLI</p>
<div class="highlight"><pre><span></span><code>prefect work-pool create my-managed-pool --type prefect:managed
<p>Create a new work pool of type Prefect Managed in the UI or the CLI.
Here's the command to create a new work pool using the CLI:</p>
<div class="terminal">
<div class="highlight"><pre><span></span><code>prefect<span class="w"> </span>work-pool<span class="w"> </span>create<span class="w"> </span>my-managed-pool<span class="w"> </span>--type<span class="w"> </span>prefect:managed
</code></pre></div>
</div>

<h3 id="step-2">Step 2<a class="headerlink" href="#step-2" title="Permanent link">&para;</a></h3>
<p>Create a deployment using the flow <code>deploy</code> method or <code>prefect.yaml</code>.</p>
<p>Specify the name of your managed work pool, as shown in this example that uses the <code>deploy</code> method:</p>
Expand All @@ -9217,7 +9223,7 @@ <h3 id="step-2">Step 2<a class="headerlink" href="#step-2" title="Permanent link
<span class="hll"> <span class="n">work_pool_name</span><span class="o">=</span><span class="s2">&quot;my-managed-pool&quot;</span><span class="p">,</span>
</span> <span class="p">)</span>
</code></pre></div>
<p>With your CLI authenticated to your Prefect Cloud workspace, run the script to create your deployment:</p>
<p>With your <a href="/cloud/users/api-keys/">CLI authenticated to your Prefect Cloud workspace</a>, run the script to create your deployment:</p>
<div class="terminal">
<div class="highlight"><pre><span></span><code>python<span class="w"> </span>managed-execution.py
</code></pre></div>
Expand All @@ -9226,9 +9232,10 @@ <h3 id="step-2">Step 2<a class="headerlink" href="#step-2" title="Permanent link
<p>Note that this deployment uses flow code stored in a GitHub repository.</p>
<h3 id="step-3">Step 3<a class="headerlink" href="#step-3" title="Permanent link">&para;</a></h3>
<p>Run the deployment from the UI or from the CLI.</p>
<p>That's it! You ran a flow on remote infrastructure without any infrastructure setup, worker, or cloud provider account.</p>
<p>That's it! You ran a flow on remote infrastructure without any infrastructure setup, starting a worker, or needing a cloud provider account.</p>
<h3 id="adding-dependencies">Adding dependencies<a class="headerlink" href="#adding-dependencies" title="Permanent link">&para;</a></h3>
<p>You can install Python package dependencies at runtime by configuring <code>job_variables={"pip_packages": ["pandas", "prefect-aws"]}</code> like this:</p>
<p>Prefect can install Python packages in the container that runs your flow at runtime.
You can specify these dependencies in the <strong>Pip Packages</strong> field in the UI, or by configuring <code>job_variables={"pip_packages": ["pandas", "prefect-aws"]}</code> in your deployment creation like this:</p>
<div class="highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">prefect</span> <span class="kn">import</span> <span class="n">flow</span>

<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&quot;__main__&quot;</span><span class="p">:</span>
Expand All @@ -9241,7 +9248,7 @@ <h3 id="adding-dependencies">Adding dependencies<a class="headerlink" href="#add
<span class="hll"> <span class="n">job_variables</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;pip_packages&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&quot;pandas&quot;</span><span class="p">,</span> <span class="s2">&quot;prefect-aws&quot;</span><span class="p">]}</span>
</span> <span class="p">)</span>
</code></pre></div>
<p>Alternatively, you can specify a <code>requirements.txt</code> file and reference it in your <code>prefect.yaml</code> <code>pull_step</code>.</p>
<p>Alternatively, you can create a <code>requirements.txt</code> file and reference it in your <code>prefect.yaml</code> <code>pull_step</code>.</p>
<h2 id="limitations">Limitations<a class="headerlink" href="#limitations" title="Permanent link">&para;</a></h2>
<p>Managed execution requires Prefect 2.14.4 or newer.</p>
<p>All limitations listed below may change without warning during the beta period.
Expand All @@ -9266,7 +9273,7 @@ <h3 id="code-storage">Code storage<a class="headerlink" href="#code-storage" tit
<h3 id="resources">Resources<a class="headerlink" href="#resources" title="Permanent link">&para;</a></h3>
<p>Memory is limited to 2GB of RAM, which includes all operations such as dependency installation. Maximum job run time is 24 hours.</p>
<h2 id="usage-limits">Usage limits<a class="headerlink" href="#usage-limits" title="Permanent link">&para;</a></h2>
<p>Free tier accounts are limited to ten compute hours per workspace per month. Pro tier and above accounts are limited to 250 hours per workspace per month. you can view your compute hours quota usage on the work pools page.</p>
<p>Free tier accounts are limited to ten compute hours per workspace per month. Pro tier and above accounts are limited to 250 hours per workspace per month. You can view your compute hours quota usage on the <strong>Work Pools</strong> page in the UI.</p>
<h2 id="next-steps">Next steps<a class="headerlink" href="#next-steps" title="Permanent link">&para;</a></h2>
<p>Read more about creating deployments in the <a href="/guides/prefect-deploy/">deployment guide</a>.</p>
<p>If you find that you need more control over your infrastructure, such as the ability to run custom Docker images, serverless push work pools might be a good option.
Expand Down
Loading

0 comments on commit 52f7910

Please sign in to comment.