Skip to content

Commit

Permalink
Trigger on job completion (#4870)
Browse files Browse the repository at this point in the history
## What are you changing in this pull request and why?

Docs for new "trigger on job completion" feature in dbt Cloud.

Updated [Deploy job](https://docs.getdbt.com/docs/deploy/deploy-jobs)
page :
- Added new section **Trigger on job completion**
- Updated screenshots, removed old ones
- Added info about the new **Triggers** section in the UI

Updated [Job
scheduler](https://docs.getdbt.com/docs/deploy/job-scheduler#scheduler-queue)
page:
- Updated section to mention "job completion"

Updated [Run
visibility](https://docs.getdbt.com/docs/deploy/run-visibility) page:
- Added new section **Run details**
- Add screenshot

New page:
- Release note

## Checklist
- [x] Review the [Content style
guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md)
so my content adheres to these guidelines.
- [x] For [docs
versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#about-versioning),
review how to [version a whole
page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version)
and [version a block of
content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content).
- [ ] Needs review from product team
  • Loading branch information
nghi-ly authored Feb 16, 2024
2 parents 165b80c + dc7b152 commit 8a04f2e
Show file tree
Hide file tree
Showing 10 changed files with 71 additions and 31 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
title: "New: Trigger on job completion"
description: "February 2024: Native support now available in dbt Cloud for triggering deploy jobs when other deploy jobs finish."
sidebar_label: "New: Trigger on job completion"
sidebar_position: 07
tags: [Feb-2024]
date: 2024-02-15
---

# New: Trigger on job completion <Lifecycle status="team,enterprise" />

Now available for dbt Cloud Team and Enterprise plans is the ability to trigger deploy jobs when other deploy jobs are complete. You can enable this feature [in the UI](/docs/deploy/deploy-jobs) with the **Run when another job finishes** option in the **Triggers** section of your job or with the [Create Job API endpoint](/dbt-cloud/api-v2#/operations/Create%20Job).

When enabled, your job will run after the specified upstream job completes. You can configure which run status(es) will trigger your job. It can be just on `Success` or on all statuses. If you have dependencies between your dbt projects, this allows you to _natively_ orchestrate your jobs within dbt Cloud &mdash; no need to set up a third-party tool.

An example of the **Triggers** section when creating the job:

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/example-triggers-section.png" width="90%" title="Example of Triggers on the Deploy Job page"/>
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "New: Keep on latest version"
description: "February 2024: New setting, called Keep on latest version, that allows dbt Labs to handle version upgrades for you."
sidebar_label: "New: Keep on latest version"
sidebar_position: 06
sidebar_position: 08
tags: [Feb-2024]
date: 2024-02-14
---
Expand Down
66 changes: 39 additions & 27 deletions website/docs/docs/deploy/deploy-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,57 +13,63 @@ You can use deploy jobs to build production data assets. Deploy jobs make it eas
- Job run details, including run timing, [model timing data](#model-timing), and [artifacts](/docs/deploy/artifacts)
- Detailed run steps with logs and their run step statuses

You can create a deploy job and configure it to run on [scheduled days and times](#schedule-days) or enter a [custom cron schedule](#custom-cron-schedules).
You can create a deploy job and configure it to run on [scheduled days and times](#schedule-days) or enter a [custom cron schedule](#cron-schedule).


## Prerequisites

- You must have a dbt Cloud account and [Developer seat license](/docs/cloud/manage-access/seats-and-users). If you don't, you can [sign up](https://www.getdbt.com/signup/) for a [free account](https://www.getdbt.com/pricing/).
- You must have a [dbt Cloud account](https://www.getdbt.com/signup/) and [Developer seat license](/docs/cloud/manage-access/seats-and-users).
- For the [Trigger on job completion](#trigger-on-job-completion) feature, your dbt Cloud account must be on the [Team or Enterprise plan](https://www.getdbt.com/pricing/).
- You must have a dbt project connected to a [data platform](/docs/cloud/connect-data-platform/about-connections).
- You must have [access permission](/docs/cloud/manage-access/about-user-access) to view, create, modify, or run jobs.
- You must set up a [deployment environment](/docs/deploy/deploy-environments).

## Create and schedule jobs {#create-and-schedule-jobs}

1. On your deployment environment page, click **Create Job** > **Deploy Job** to create a new deploy job.
2. Options in the **Job Description** section:
- **Job Name** &mdash; Specify the name for the deploy job. For example, `Daily build`.
1. On your deployment environment page, click **Create job** > **Deploy job** to create a new deploy job.
2. Options in the **Job settings** section:
- **Job name** &mdash; Specify the name for the deploy job. For example, `Daily build`.
- (Optional) **Description** &mdash; Provide a description of what the job does (for example, what the job consumes and what the job produces).
- **Environment** &mdash; By default, it’s set to the deployment environment you created the deploy job from.
3. Options in the **Execution Settings** section:
3. Options in the **Execution settings** section:
- **Commands** &mdash; By default, it includes the `dbt build` command. Click **Add command** to add more [commands](/docs/deploy/job-commands) that you want to be invoked when the job runs.
- **Generate docs on run** &mdash; Enable this option if you want to [generate project docs](/docs/collaborate/build-and-view-your-docs) when this deploy job runs.
- **Run source freshness** &mdash; Enable this option to invoke the `dbt source freshness` command before running the deploy job. Refer to [Source freshness](/docs/deploy/source-freshness) for more details.
4. Options in the **Schedule** section:
- **Run on schedule** &mdash; Enable this option to run the deploy job on a set schedule.
- **Timing** &mdash; Specify whether to [schedule](#schedule-days) the deploy job using **Frequency** that runs the job at specific times of day, **Specific Intervals** that runs the job every specified number of hours, or **Cron Schedule** that runs the job specified using [cron syntax](#custom-cron-schedule).
- **Days of the Week** &mdash; By default, it’s set to every day when **Frequency** or **Specific Intervals** is chosen for **Timing**.

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/create-deploy-job.png" width="90%" title="Example of Deploy Job page in dbt Cloud UI"/>

5. (optional) Options in the **Advanced Settings** section:
- **Environment Variables** &mdash; Define [environment variables](/docs/build/environment-variables) to customize the behavior of your project when the deploy job runs.
- **Target Name** &mdash; Define the [target name](/docs/build/custom-target-names) to customize the behavior of your project when the deploy job runs. Environment variables and target names are often used interchangeably.
- **Run Timeout** &mdash; Cancel the deploy job if the run time exceeds the timeout value.
4. Options in the **Triggers** section:
- **Run on schedule** &mdash; Run the deploy job on a set schedule.
- **Timing** &mdash; Specify whether to [schedule](#schedule-days) the deploy job using **Hours of the day** that runs the job at specific times of day, **Exact intervals** that runs the job every specified number of hours, or **Cron schedule** that runs the job specified using [cron syntax](#cron-schedule).
- **Days of the week** &mdash; By default, it’s set to every day when **Hours of the day** or **Exact intervals** is chosen for **Timing**.
- **Run when another job finishes** &mdash; Run the deploy job when another _upstream_ deploy [job completes](#trigger-on-job-completion).
- **Project** &mdash; Specify the parent project that has that upstream deploy job.
- **Job** &mdash; Specify the upstream deploy job.
- **Completes on** &mdash; Select the job run status(es) that will [enqueue](/docs/deploy/job-scheduler#scheduler-queue) the deploy job.

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/example-triggers-section.png" width="90%" title="Example of Triggers on the Deploy Job page"/>

5. (Optional) Options in the **Advanced settings** section:
- **Environment variables** &mdash; Define [environment variables](/docs/build/environment-variables) to customize the behavior of your project when the deploy job runs.
- **Target name** &mdash; Define the [target name](/docs/build/custom-target-names) to customize the behavior of your project when the deploy job runs. Environment variables and target names are often used interchangeably.
- **Run timeout** &mdash; Cancel the deploy job if the run time exceeds the timeout value.
- **Compare changes against** &mdash; By default, it’s set to **No deferral**. Select either **Environment** or **This Job** to let dbt Cloud know what it should compare the changes against.

:::info
Older versions of dbt Cloud only allow you to defer to a specific job instead of an environment. Deferral to a job compares state against the project code that was run in the deferred job's last successful run. While deferral to an environment is more efficient as dbt Cloud will compare against the project representation (which is stored in the `manifest.json`) of the last successful deploy job run that executed in the deferred environment. By considering _all_ deploy jobs that run in the deferred environment, dbt Cloud will get a more accurate, latest project representation state.
:::

- **dbt Version** &mdash; By default, it’s set to inherit the [dbt version](/docs/dbt-versions/core) from the environment. dbt Labs strongly recommends that you don't change the default setting. This option to change the version at the job level is useful only when you upgrade a project to the next dbt version; otherwise, mismatched versions between the environment and job can lead to confusing behavior.
- **dbt version** &mdash; By default, it’s set to inherit the [dbt version](/docs/dbt-versions/core) from the environment. dbt Labs strongly recommends that you don't change the default setting. This option to change the version at the job level is useful only when you upgrade a project to the next dbt version; otherwise, mismatched versions between the environment and job can lead to confusing behavior.
- **Threads** &mdash; By default, it’s set to 4 [threads](/docs/core/connect-data-platform/connection-profiles#understanding-threads). Increase the thread count to increase model execution concurrency.

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/deploy-job-adv-settings.png" width="90%" title="Example of Advanced Settings on Deploy Job page"/>
<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/deploy-job-adv-settings.png" width="90%" title="Example of Advanced Settings on the Deploy Job page"/>

### Schedule days

To set your job's schedule, use the **Schedule Days** option to choose specific days of the week, and select customized hours or intervals.
To set your job's schedule, use the **Run on schedule** option to choose specific days of the week, and select customized hours or intervals.

Under **Timing**, you can either use customizable hours for jobs that need to run frequently throughout the day or exact intervals for jobs that need to run at specific times:

- **Every n hours** &mdash; Use this option to set how often your job runs, in hours. Enter a number between 1 and 23 to represent the interval between job runs. For example, if you set it to "every 2 hours", the job will run every 2 hours from midnight UTC. This option is useful if you need to run jobs multiple times per day at regular intervals.
- **Exact intervals** &mdash; Use this option to set how often your job runs, in hours. Enter a number between 1 and 23 to represent the interval between job runs. For example, if you set it to **Every 2 hours**, the job will run every 2 hours from midnight UTC. This option is useful if you need to run jobs multiple times per day at regular intervals.

- **At exact intervals** &mdash; Use this option to set specific times when your job should run. You can enter a comma-separated list of hours (in UTC) when you want the job to run. For example, if you set it to `0,12,23,` the job will run at midnight, noon, and 11 PM UTC. This option is useful if you want your jobs to run at specific times of day and don't need them to run more frequently than once a day.
- **Hours of the day** &mdash; Use this option to set specific times when your job should run. You can enter a comma-separated list of hours (in UTC) when you want the job to run. For example, if you set it to `0,12,23,` the job will run at midnight, noon, and 11 PM UTC. This option is useful if you want your jobs to run at specific times of day and don't need them to run more frequently than once a day.

:::info

Expand All @@ -75,12 +81,9 @@ dbt Cloud uses [Coordinated Universal Time](https://en.wikipedia.org/wiki/Coordi

:::

### Custom cron schedule

To fully customize the scheduling of your job, choose the **Custom cron schedule** option and use the cron syntax. With this syntax, you can specify the minute, hour, day of the month, month, and day of the week, allowing you to set up complex schedules like running a job on the first Monday of each month.
### Cron schedule


<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/job-schedule.png" title="Schedule your dbt job"/>
To fully customize the scheduling of your job, choose the **Cron schedule** option and use cron syntax. With this syntax, you can specify the minute, hour, day of the month, month, and day of the week, allowing you to set up complex schedules like running a job on the first Monday of each month.

Use tools such as [crontab.guru](https://crontab.guru/) to generate the correct cron syntax. This tool allows you to input cron snippets and returns their plain English translations.

Expand All @@ -101,6 +104,15 @@ Here are examples of cron job schedules. The dbt Cloud job scheduler supports us
- `0 7 L * 5`: At 07:00 AM, on the last day of the month, and on Friday.
- `30 14 L * *`: At 02:30 PM, on the last day of the month.

### Trigger on job completion <Lifecycle status="team,enterprise" />

To _chain_ deploy jobs together, enable the **Run when another job finishes** option and specify the upstream (parent) job that, when it completes, will trigger your job. You can also use the [Create Job API](/dbt-cloud/api-v2#/operations/Create%20Job) to do this.

You can set up a configuration where an upstream job triggers multiple downstream (child) jobs and jobs in other projects. You must have proper [permissions](/docs/cloud/manage-access/enterprise-permissions#project-role-permissions) to the project and job to configure the trigger.

For jobs that are triggered to run by another job, a link to the upstream job run is available from your [job's run details](/docs/deploy/run-visibility#job-run-details).


## Related docs

- [Artifacts](/docs/deploy/artifacts)
Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/deploy/job-scheduler.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ Familiarize yourself with these useful terms to help you understand how the job

## Scheduler queue

The scheduler queues a deployment job to be processed when it's triggered to run because of a [set schedule](#create-and-schedule-jobs), an API call, or manual action.
The scheduler queues a deployment job to be processed when it's triggered to run by a [set schedule](/docs/deploy/deploy-jobs#schedule-days), [a job completed](/docs/deploy/deploy-jobs#trigger-on-job-completion), an API call, or manual action.

Before the job starts executing, the scheduler checks these conditions to determine if the run can start executing:

Expand Down
14 changes: 12 additions & 2 deletions website/docs/docs/deploy/run-visibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,25 @@ You can view the history of your runs and the model timing dashboard to help ide

## Run history

The **Run History** dashboard in dbt Cloud helps you monitor the health of your dbt project. It provides a detailed overview of all of your project's job runs and empowers you with a variety of filters to help you focus on specific aspects. You can also use it to review recent runs, find errored runs, and track the progress of runs in progress. You can access it on the top navigation menu by clicking **Deploy** and then **Run History**.
The **Run history** dashboard in dbt Cloud helps you monitor the health of your dbt project. It provides a detailed overview of all of your project's job runs and empowers you with a variety of filters to help you focus on specific aspects. You can also use it to review recent runs, find errored runs, and track the progress of runs in progress. You can access it on the top navigation menu by clicking **Deploy** and then **Run history**.

The dashboard displays your full run history, including job name, status, associated environment, job trigger, commit SHA, schema, and timing info.

dbt Cloud developers can access their run history for the last 365 days through the dbt Cloud user interface (UI) and API.

We limit self-service retrieval of run history metadata to 365 days to improve dbt Cloud's performance. For more info on the run history retrieval change, refer to [Older run history retrieval change](/docs/dbt-versions/release-notes/May-2023/run-history-endpoint).

<Lightbox src="/img/docs/dbt-cloud/deployment/run-history.jpg" width="85%" title="Run History dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more."/>
<Lightbox src="/img/docs/dbt-cloud/deployment/run-history.jpg" width="85%" title="Run history dashboard allows you to monitor the health of your dbt project and displays jobs, job status, environment, timing, and more."/>

## Job run details

From the **Run history** dashboard, select a run to view complete details about it. The job run details page displays job trigger, commit SHA, time spent in the scheduler queue, all the run steps and their [logs](#access-logs), [model timing](#model-timing), and more.

Click **Rerun now** to rerun the job immediately.

An example of a completed run with a configuration for a [job completion trigger](/docs/deploy/deploy-jobs#trigger-on-job-completion):

<Lightbox src="/img/docs/dbt-cloud/deployment/example-job-details.png" width="70%" title="Example of run details" />

## Access logs

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.

0 comments on commit 8a04f2e

Please sign in to comment.