Skip to content

Commit

Permalink
[docs] - Make new Schedules page public (#22331)
Browse files Browse the repository at this point in the history
## Summary & Motivation

This PR:

- Makes the revamped Schedules content public and deprecates (removes)
the old page (`/concepts/partitions-schedules-sensors/schedules`)
- Updates links in the docs. **Note**: Only links in docs content are
updated in this PR - links in the UI and elsewhere in the repo will be
in a subsequent PR.
- Adds redirects

## How I Tested These Changes

local, testing redirects
  • Loading branch information
erinkcochran87 authored Jun 7, 2024
1 parent a804e08 commit 640a976
Show file tree
Hide file tree
Showing 40 changed files with 97 additions and 616 deletions.
3 changes: 1 addition & 2 deletions docs/content/_apidocs.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,7 @@ APIs from the core `dagster` package, divided roughly by topic:
<a href="/_apidocs/schedules-sensors">Schedules & Sensors</a>
</td>
<td>
APIs to define{" "}
<a href="/concepts/partitions-schedules-sensors/schedules">schedules</a>{" "}
APIs to define <a href="/concepts/automation/schedules">schedules</a>{" "}
and <a href="/concepts/partitions-schedules-sensors/sensors">sensors</a>{" "}
that initiate job execution, as well as some built-in helpers for common
cases.
Expand Down
36 changes: 35 additions & 1 deletion docs/content/_navigation.json
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,41 @@
},
{
"title": "Schedules",
"path": "/concepts/partitions-schedules-sensors/schedules"
"path": "/concepts/automation/schedules",
"children": [
{
"title": "Overview",
"path": "/concepts/automation/schedules"
},
{
"title": "Automating assets",
"path": "/concepts/automation/schedules/automating-assets-schedules-jobs"
},
{
"title": "Automating ops",
"path": "/concepts/automation/schedules/automating-ops-schedules-jobs"
},
{
"title": "Examples",
"path": "/concepts/automation/schedules/examples"
},
{
"title": "Partitioned schedules",
"path": "/concepts/automation/schedules/partitioned-schedules"
},
{
"title": "Customizing timezones",
"path": "/concepts/automation/schedules/customizing-executing-timezones"
},
{
"title": "Testing",
"path": "/concepts/automation/schedules/testing"
},
{
"title": "Troubleshooting",
"path": "/concepts/automation/schedules/troubleshooting"
}
]
},
{
"title": "Sensors",
Expand Down
2 changes: 1 addition & 1 deletion docs/content/concepts.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ Dagster offers several ways to run data pipelines without manual intervention, i
></ArticleListItem>
<ArticleListItem
title="Schedules"
href="/concepts/partitions-schedules-sensors/schedules"
href="/concepts/automation/schedules"
></ArticleListItem>
<ArticleListItem
title="Sensors"
Expand Down
2 changes: 1 addition & 1 deletion docs/content/concepts/assets/asset-auto-execution.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ def asset1(): ...
def asset2(): ...
```

This example assumes that `asset1` will be materialized in some other way - e.g. manually, via a [sensor](/concepts/partitions-schedules-sensors/sensors), or via a [schedule](/concepts/partitions-schedules-sensors/schedules).
This example assumes that `asset1` will be materialized in some other way - e.g. manually, via a [sensor](/concepts/partitions-schedules-sensors/sensors), or via a [schedule](/concepts/automation/schedules).

### Adding an auto-materialize policy to multiple assets at once

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Before continuing, you should be familiar with:

- [Asset definitions](/concepts/assets/software-defined-assets)
- [Asset checks](/concepts/assets/asset-checks)
- [Schedules](/concepts/partitions-schedules-sensors/schedules) and/or [sensors](/concepts/partitions-schedules-sensors/sensors)
- [Schedules](/concepts/automation/schedules) and/or [sensors](/concepts/partitions-schedules-sensors/sensors)
- [Alerts](/dagster-plus/managing-deployments/alerts) (Dagster+ only)

---
Expand Down Expand Up @@ -91,7 +91,7 @@ def source_tables(snowflake: SnowflakeResource):

### Step 2: Schedule the observations

Next, we'll define a [schedule](/concepts/partitions-schedules-sensors/schedules) that regularly executes the function in the `source_tables` observable source asset:
Next, we'll define a [schedule](/concepts/automation/schedules) that regularly executes the function in the `source_tables` observable source asset:

```python file=/concepts/assets/asset_checks/source_data_freshness_in_pieces.py startafter=start_schedule_marker endbefore=end_schedule_marker
from dagster import AssetSelection, ScheduleDefinition, define_asset_job
Expand Down Expand Up @@ -280,7 +280,7 @@ asset1_freshness_checks = build_last_update_freshness_checks(

### Step 2: Create a sensor to execute the check

A [schedule](/concepts/partitions-schedules-sensors/schedules) or [sensor](/concepts/partitions-schedules-sensors/sensors) is required to ensure the freshness check executes. If the check only runs after the asset has been materialized, the check won't be able to detect the times materialization fails.
A [schedule](/concepts/automation/schedules) or [sensor](/concepts/partitions-schedules-sensors/sensors) is required to ensure the freshness check executes. If the check only runs after the asset has been materialized, the check won't be able to detect the times materialization fails.

In this example, we'll use <PyObject object="build_sensor_for_freshness_checks" /> to create a sensor to automatically run the check. Based on the check's parameters and last run time, the sensor will run the check when enough time has elapsed that the asset might fail the check:

Expand Down Expand Up @@ -396,7 +396,7 @@ To alert on overdue assets, create an alert policy with the following settings:
></ArticleListItem>
<ArticleListItem
title="Schedules"
href="/concepts/partitions-schedules-sensors/schedules"
href="/concepts/automation/schedules"
></ArticleListItem>
<ArticleListItem
title="Sensors"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -331,7 +331,7 @@ Materializing an asset from the UI will also execute any checks defined for that

### Via sensors and schedules

To define jobs that execute sets of assets and checks, you can use <PyObject object="define_asset_job" /> and then trigger the jobs via [sensors](/concepts/partitions-schedules-sensors/sensors) or [schedules](/concepts/partitions-schedules-sensors/schedules). By default, checks are included with the assets they check. You can also define jobs that include only checks, or only assets.
To define jobs that execute sets of assets and checks, you can use <PyObject object="define_asset_job" /> and then trigger the jobs via [sensors](/concepts/partitions-schedules-sensors/sensors) or [schedules](/concepts/automation/schedules). By default, checks are included with the assets they check. You can also define jobs that include only checks, or only assets.

```python file=/concepts/assets/asset_checks/jobs.py
from dagster import (
Expand Down Expand Up @@ -434,7 +434,7 @@ In Dagster+, you can set up alerts to notify you when assets checks fail. To ale
></ArticleListItem>
<ArticleListItem
title="Schedules"
href="/concepts/partitions-schedules-sensors/schedules"
href="/concepts/automation/schedules"
></ArticleListItem>
<ArticleListItem
title="Sensors"
Expand Down
2 changes: 1 addition & 1 deletion docs/content/concepts/assets/asset-jobs.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ description: Asset jobs are the main unit for materializing and monitoring asset
[Jobs](/concepts/ops-jobs-graphs/jobs) are the main unit for executing and monitoring [asset definitions](/concepts/assets/software-defined-assets) in Dagster. An asset job is a type of \[job]\(/concepts/ops-jobs-graphs/jobs] that targets a selection of assets and can be launched:

- Manually from the Dagster UI
- At fixed intervals, by [schedules](/concepts/partitions-schedules-sensors/schedules)
- At fixed intervals, by [schedules](/concepts/automation/schedules)
- When external changes occur, using [sensors](/concepts/partitions-schedules-sensors/sensors)

---
Expand Down
4 changes: 2 additions & 2 deletions docs/content/concepts/automation.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ The following cheatsheet contains high-level details about each of the automatio
<tbody>
<tr>
<td>
<a href="/concepts/partitions-schedules-sensors/schedules">Schedules</a>
<a href="/concepts/automation/schedules">Schedules</a>
</td>
<td>Starts a job at a specified time</td>
<td>
Expand Down Expand Up @@ -245,7 +245,7 @@ The following cheatsheet contains high-level details about each of the automatio

[op-jobs]: /concepts/ops-jobs-graphs/op-jobs

[schedules]: /concepts/partitions-schedules-sensors/schedules
[schedules]: /concepts/automation/schedules

[sensors]: /concepts/partitions-schedules-sensors/sensors

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: "By default, schedules without a set timezone execute in UTC. Learn

# Customizing a schedule's executing timezone

[Schedules](/concepts/partitions-schedules-sensors/schedules) that don't have a set timezone will, by default, execute in [UTC](https://en.wikipedia.org/wiki/Coordinated_Universal_Time). By the end of this guide, you'll know how to:
[Schedules](/concepts/automation/schedules) that don't have a set timezone will, by default, execute in [UTC](https://en.wikipedia.org/wiki/Coordinated_Universal_Time). By the end of this guide, you'll know how to:

- Set custom timezones on schedule definitions
- Set custom timezones on partitioned jobs
Expand All @@ -17,7 +17,7 @@ description: "By default, schedules without a set timezone execute in UTC. Learn

To follow this guide, you need to be familiar with:

- [Schedules](/concepts/partitions-schedules-sensors/schedules)
- [Schedules](/concepts/automation/schedules)
- Jobs, either [asset](/concepts/assets/asset-jobs) or [op-based](/concepts/ops-jobs-graphs/op-jobs)
- [Partitions](/concepts/partitions-schedules-sensors/partitions) (Optional)

Expand Down
2 changes: 1 addition & 1 deletion docs/content/concepts/automation/schedules/examples.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: "Examples focused on Dagster schedules."

# Schedule examples

This reference contains a variety of examples using Dagster [schedules](/concepts/partitions-schedules-sensors/schedules). Each example contains:
This reference contains a variety of examples using Dagster [schedules](/concepts/automation/schedules). Each example contains:

- A summary
- Additional notes
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ In this guide, we'll walk you through how to construct schedules from partitione

To follow this guide, you need to be familiar with:

- [Schedules](/concepts/partitions-schedules-sensors/schedules)
- [Schedules](/concepts/automation/schedules)
- [Partitions](/concepts/partitions-schedules-sensors/partitions)
- [Asset definitions](/concepts/assets/software-defined-assets)
- [Asset jobs](/concepts/assets/asset-jobs) and [op jobs](/concepts/ops-jobs-graphs/op-jobs)
Expand Down Expand Up @@ -243,7 +243,7 @@ def antarctica_schedule():
<ArticleList>
<ArticleListItem
title="Schedules"
href="/concepts/partitions-schedules-sensors/schedules"
href="/concepts/automation/schedules"
></ArticleListItem>
<ArticleListItem
title="Partitions"
Expand Down
2 changes: 1 addition & 1 deletion docs/content/concepts/automation/schedules/testing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: "Learn to test your schedules in Python or the Dagster UI."

# Testing schedules

In this guide, we'll show you how to use the Dagster UI and Python to test your [schedules](/concepts/partitions-schedules-sensors/schedules). For more information about testing in Dagster, refer to the [Testing documentation](/concepts/testing).
In this guide, we'll show you how to use the Dagster UI and Python to test your [schedules](/concepts/automation/schedules). For more information about testing in Dagster, refer to the [Testing documentation](/concepts/testing).

---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ The last step is to check that the schedule executed successfully. If you didn't
<ArticleList>
<ArticleListItem
title="Schedules"
href="/concepts/partitions-schedules-sensors/schedules"
href="/concepts/automation/schedules"
></ArticleListItem>
<ArticleListItem
title="Code locations"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ It's often useful to provide user-chosen values to Dagster jobs or asset definit

Various Dagster entities (ops, assets, resources) can be individually configured. When launching a job that executes (ops), materializes (assets), or instantiates (resources) a configurable entity, you can provide _run configuration_ for each entity. Within the function that defines the entity, you can access the passed-in configuration off of the `context`. Typically, the provided run configuration values correspond to a _configuration schema_ attached to the op/asset/resource definition. Dagster validates the run configuration against the schema and proceeds only if validation is successful.

A common use of configuration is for a [schedule](/concepts/partitions-schedules-sensors/schedules) or [sensor](/concepts/partitions-schedules-sensors/schedules) to provide configuration to the job run it is launching. For example, a daily schedule might provide the day it's running on to one of the ops as a config value, and that op might use that config value to decide what day's data to read.
A common use of configuration is for a [schedule](/concepts/automation/schedules) or [sensor](/concepts/partitions-schedules-sensors/sensors) to provide configuration to the job run it is launching. For example, a daily schedule might provide the day it's running on to one of the ops as a config value, and that op might use that config value to decide what day's data to read.

---

Expand Down
2 changes: 1 addition & 1 deletion docs/content/concepts/configuration/config-schema.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ It's often useful to provide user-chosen values to Dagster jobs or asset definit

Various Dagster entities (assets, ops, resources) can be individually configured. When launching a job that materializes (assets), executes (ops), or instantiates (resources) a configurable entity, you can provide _run configuration_ for each entity. Within the function that defines the entity, you can access the passed-in configuration through the `config` parameter. Typically, the provided run configuration values correspond to a _configuration schema_ attached to the asset/op/resource definition. Dagster validates the run configuration against the schema and proceeds only if validation is successful.

A common use of configuration is for a [schedule](/concepts/partitions-schedules-sensors/schedules) or [sensor](/concepts/partitions-schedules-sensors/schedules) to provide configuration to the job run it is launching. For example, a daily schedule might provide the day it's running on to one of the assets as a config value, and that asset might use that config value to decide what day's data to read.
A common use of configuration is for a [schedule](/concepts/automation/schedules) or [sensor](/concepts/partitions-schedules-sensors/sensors) to provide configuration to the job run it is launching. For example, a daily schedule might provide the day it's running on to one of the assets as a config value, and that asset might use that config value to decide what day's data to read.

---

Expand Down
2 changes: 1 addition & 1 deletion docs/content/concepts/ops-jobs-graphs/job-execution.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Dagster provides several methods to execute [op](/concepts/ops-jobs-graphs/op-jo

You can also launch jobs in other ways:

- [Schedules](/concepts/partitions-schedules-sensors/schedules) can be used to launch runs on a fixed interval.
- [Schedules](/concepts/automation/schedules) can be used to launch runs on a fixed interval.
- [Sensors](/concepts/partitions-schedules-sensors/sensors) allow you to launch runs based on external state changes.

---
Expand Down
2 changes: 1 addition & 1 deletion docs/content/concepts/ops-jobs-graphs/jobs.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ When a job begins, it kicks off a _run_. A run is a single execution of a job in

Using jobs provides the following benefits:

- **Automation**: With [schedules](/concepts/partitions-schedules-sensors/schedules) and [sensors](/concepts/partitions-schedules-sensors/sensors), jobs can be used to automate the execution of your Dagster pipelines. Refer to the [Automation guide](/concepts/automation) for more info.
- **Automation**: With [schedules](/concepts/automation/schedules) and [sensors](/concepts/partitions-schedules-sensors/sensors), jobs can be used to automate the execution of your Dagster pipelines. Refer to the [Automation guide](/concepts/automation) for more info.
- **Control job run priority**: If using a [run queue](/deployment/run-coordinator), you can apply custom prioritization rules to how job runs are prioritized and executed.
- **Potential for improved efficency**: By applying concurrency limits to job runs, there may be benefits to your pipeline's efficiency. Refer to the [Limiting run concurrency guide](/guides/limiting-concurrency-in-data-pipelines) for more info and examples.

Expand Down
2 changes: 1 addition & 1 deletion docs/content/concepts/ops-jobs-graphs/op-jobs.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ description: An op job
Op jobs can be launched in a few different ways:

- Manually from the Dagster UI
- At fixed intervals, by [schedules](/concepts/partitions-schedules-sensors/schedules)
- At fixed intervals, by [schedules](/concepts/automation/schedules)
- When external changes occur, using [sensors](/concepts/partitions-schedules-sensors/sensors)

---
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ def continent_job():

Running a partitioned job on a schedule is a common use case. For example, if your job has a partition for each date, you likely want to run that job every day, on the partition for that day.

Refer to the [Schedule documentation](/concepts/partitions-schedules-sensors/schedules#schedules-from-partitioned-assets-and-jobs) for more info about constructing both schedules for asset and op-based jobs.
Refer to the [Schedule documentation](/concepts/automation/schedules/partitioned-schedules) for more info about constructing both schedules for asset and op-based jobs.

---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,5 @@ Partitions are supported for both asset definitions and ops, but how each concep
With partitions, you can:

- View runs by partition in the [Dagster UI](/concepts/webserver/ui)
- Define a [schedule](/concepts/partitions-schedules-sensors/schedules) that fills in a partition each time it runs. For example, a job might run each day and process the data that arrived during the previous day.
- Define a [schedule](/concepts/automation/schedules) that fills in a partition each time it runs. For example, a job might run each day and process the data that arrived during the previous day.
- Launch [backfills](/concepts/partitions-schedules-sensors/backfills), which are sets of runs that each process a different partition. For example, after making a code change, you might want to run your job on all partitions instead of just one of them.
Loading

1 comment on commit 640a976

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs ready!

✅ Preview
https://dagster-docs-3y71doxh9-elementl.vercel.app
https://master.dagster.dagster-docs.io

Built with commit 640a976.
This pull request is being automatically deployed with vercel-action

Please sign in to comment.