Skip to content

Commit

Permalink
Merge branch 'current' into fix/lightbox-hover
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Feb 16, 2024
2 parents 5852a9b + 8a04f2e commit 703705a
Show file tree
Hide file tree
Showing 11 changed files with 76 additions and 44 deletions.
18 changes: 5 additions & 13 deletions website/docs/docs/collaborate/explore-projects.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,22 +75,16 @@ To interact with the full lineage graph, you can:


## Search for resources {#search-resources}
With the search bar (on the upper left corner of the page or in a lineage graph), you can search with keywords or by using [node selection syntax](/reference/node-selection/syntax). The resources that match your search criteria will display as a lineage graph and a table in the main section of the page.

Select a node (single-click) in the lineage graph to highlight its relationship with your other search results and to display which project contains the resource's definition. When you choose a node (double-click) in the lineage graph or when you select a resource in the table, dbt Explorer displays the [resource's details page](#view-resource-details).
With the search bar (on the upper left corner of the page or in a lineage graph), you can search with keywords or selectors. The resources that match your search criteria will display in a list (keyword search) or the lineage graph (selector search).

### Search with keywords
When searching with keywords, dbt Explorer searches through your resource metadata (such as resource type, resource name, column name, source name, tags, schema, database, version, alias/identifier, and package name) and returns any matches.

- Keyword search features a side panel (to the right of the main section) to filter search results by resource type.
- Use this panel to select specific resource tags or model access levels under the **Models** option.
- For example, a search for "sale" returns results that include all resources with the keyword "sale" in their metadata. Filtering by **Models** and **Sources** refines these results to only include models or sources.
When searching with keywords, dbt Explorer searches through your resource metadata, such as resource name, and returns any matches. When searching for an exact column name, the results show all relational nodes containing that column in their schemas. If there's a match, a notice in the search result indicates the resource contains the specified column.

- When searching for an exact column name, the results show all relational nodes containing that column in their schemas. If there's a match, a notice in the search result indicates the resource contains the specified column.
You can then refine the search results using the filters in the panel on the right-hand side of the page. Use this panel to select specific resource types or tags to apply. You can filter on model properties (access or materialization type) under the **Models** option.

### Search with selectors

You can search with [selectors](/reference/node-selection/methods). Below are the selectors currently available in dbt Explorer:
You can search with [selectors](/reference/node-selection/methods) within the full lineage graph using the search bar. Below are the selectors currently available in dbt Explorer:

- `fqn:` — Find resources by [file or fully qualified name](/reference/node-selection/methods#the-fqn-method). This selector is the search bar's default. If you want to use the default, it's unnecessary to add `fqn:` before the search term.
- `source:` — Find resources by a specified [source](/reference/node-selection/methods#the-source-method).
Expand All @@ -105,9 +99,7 @@ You can search with [selectors](/reference/node-selection/methods). Below are th

</VersionBlock>

Because the results of selectors are immutable, the filter side panel is not available with this search method.

When searching with selector methods, you can also use [graph operators](/reference/node-selection/graph-operators). For example, `+orders` returns all the parents of `orders`. This functionality is not available for keyword search.
When searching with selector methods, you can also use [graph operators](/reference/node-selection/graph-operators). For example, `+orders` returns all the upstream nodes of `orders` and `orders+1` returns just the nodes immediately downstream (the children) of `orders`.

You can use multiple selector methods in your search query with [set operators](/reference/node-selection/set-operators). A space implies a union set operator and a comma for an intersection. For example:
- `resource_type:metric,tag:nightly` &mdash; Returns metrics with the tag `nightly`
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
title: "New: Trigger on job completion"
description: "February 2024: Native support now available in dbt Cloud for triggering deploy jobs when other deploy jobs finish."
sidebar_label: "New: Trigger on job completion"
sidebar_position: 07
tags: [Feb-2024]
date: 2024-02-15
---

# New: Trigger on job completion <Lifecycle status="team,enterprise" />

Now available for dbt Cloud Team and Enterprise plans is the ability to trigger deploy jobs when other deploy jobs are complete. You can enable this feature [in the UI](/docs/deploy/deploy-jobs) with the **Run when another job finishes** option in the **Triggers** section of your job or with the [Create Job API endpoint](/dbt-cloud/api-v2#/operations/Create%20Job).

When enabled, your job will run after the specified upstream job completes. You can configure which run status(es) will trigger your job. It can be just on `Success` or on all statuses. If you have dependencies between your dbt projects, this allows you to _natively_ orchestrate your jobs within dbt Cloud &mdash; no need to set up a third-party tool.

An example of the **Triggers** section when creating the job:

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/example-triggers-section.png" width="90%" title="Example of Triggers on the Deploy Job page"/>
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "New: Keep on latest version"
description: "February 2024: New setting, called Keep on latest version, that allows dbt Labs to handle version upgrades for you."
sidebar_label: "New: Keep on latest version"
sidebar_position: 06
sidebar_position: 08
tags: [Feb-2024]
date: 2024-02-14
---
Expand Down
66 changes: 39 additions & 27 deletions website/docs/docs/deploy/deploy-jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,57 +13,63 @@ You can use deploy jobs to build production data assets. Deploy jobs make it eas
- Job run details, including run timing, [model timing data](#model-timing), and [artifacts](/docs/deploy/artifacts)
- Detailed run steps with logs and their run step statuses

You can create a deploy job and configure it to run on [scheduled days and times](#schedule-days) or enter a [custom cron schedule](#custom-cron-schedules).
You can create a deploy job and configure it to run on [scheduled days and times](#schedule-days) or enter a [custom cron schedule](#cron-schedule).


## Prerequisites

- You must have a dbt Cloud account and [Developer seat license](/docs/cloud/manage-access/seats-and-users). If you don't, you can [sign up](https://www.getdbt.com/signup/) for a [free account](https://www.getdbt.com/pricing/).
- You must have a [dbt Cloud account](https://www.getdbt.com/signup/) and [Developer seat license](/docs/cloud/manage-access/seats-and-users).
- For the [Trigger on job completion](#trigger-on-job-completion) feature, your dbt Cloud account must be on the [Team or Enterprise plan](https://www.getdbt.com/pricing/).
- You must have a dbt project connected to a [data platform](/docs/cloud/connect-data-platform/about-connections).
- You must have [access permission](/docs/cloud/manage-access/about-user-access) to view, create, modify, or run jobs.
- You must set up a [deployment environment](/docs/deploy/deploy-environments).

## Create and schedule jobs {#create-and-schedule-jobs}

1. On your deployment environment page, click **Create Job** > **Deploy Job** to create a new deploy job.
2. Options in the **Job Description** section:
- **Job Name** &mdash; Specify the name for the deploy job. For example, `Daily build`.
1. On your deployment environment page, click **Create job** > **Deploy job** to create a new deploy job.
2. Options in the **Job settings** section:
- **Job name** &mdash; Specify the name for the deploy job. For example, `Daily build`.
- (Optional) **Description** &mdash; Provide a description of what the job does (for example, what the job consumes and what the job produces).
- **Environment** &mdash; By default, it’s set to the deployment environment you created the deploy job from.
3. Options in the **Execution Settings** section:
3. Options in the **Execution settings** section:
- **Commands** &mdash; By default, it includes the `dbt build` command. Click **Add command** to add more [commands](/docs/deploy/job-commands) that you want to be invoked when the job runs.
- **Generate docs on run** &mdash; Enable this option if you want to [generate project docs](/docs/collaborate/build-and-view-your-docs) when this deploy job runs.
- **Run source freshness** &mdash; Enable this option to invoke the `dbt source freshness` command before running the deploy job. Refer to [Source freshness](/docs/deploy/source-freshness) for more details.
4. Options in the **Schedule** section:
- **Run on schedule** &mdash; Enable this option to run the deploy job on a set schedule.
- **Timing** &mdash; Specify whether to [schedule](#schedule-days) the deploy job using **Frequency** that runs the job at specific times of day, **Specific Intervals** that runs the job every specified number of hours, or **Cron Schedule** that runs the job specified using [cron syntax](#custom-cron-schedule).
- **Days of the Week** &mdash; By default, it’s set to every day when **Frequency** or **Specific Intervals** is chosen for **Timing**.

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/create-deploy-job.png" width="90%" title="Example of Deploy Job page in dbt Cloud UI"/>

5. (optional) Options in the **Advanced Settings** section:
- **Environment Variables** &mdash; Define [environment variables](/docs/build/environment-variables) to customize the behavior of your project when the deploy job runs.
- **Target Name** &mdash; Define the [target name](/docs/build/custom-target-names) to customize the behavior of your project when the deploy job runs. Environment variables and target names are often used interchangeably.
- **Run Timeout** &mdash; Cancel the deploy job if the run time exceeds the timeout value.
4. Options in the **Triggers** section:
- **Run on schedule** &mdash; Run the deploy job on a set schedule.
- **Timing** &mdash; Specify whether to [schedule](#schedule-days) the deploy job using **Hours of the day** that runs the job at specific times of day, **Exact intervals** that runs the job every specified number of hours, or **Cron schedule** that runs the job specified using [cron syntax](#cron-schedule).
- **Days of the week** &mdash; By default, it’s set to every day when **Hours of the day** or **Exact intervals** is chosen for **Timing**.
- **Run when another job finishes** &mdash; Run the deploy job when another _upstream_ deploy [job completes](#trigger-on-job-completion).
- **Project** &mdash; Specify the parent project that has that upstream deploy job.
- **Job** &mdash; Specify the upstream deploy job.
- **Completes on** &mdash; Select the job run status(es) that will [enqueue](/docs/deploy/job-scheduler#scheduler-queue) the deploy job.

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/example-triggers-section.png" width="90%" title="Example of Triggers on the Deploy Job page"/>

5. (Optional) Options in the **Advanced settings** section:
- **Environment variables** &mdash; Define [environment variables](/docs/build/environment-variables) to customize the behavior of your project when the deploy job runs.
- **Target name** &mdash; Define the [target name](/docs/build/custom-target-names) to customize the behavior of your project when the deploy job runs. Environment variables and target names are often used interchangeably.
- **Run timeout** &mdash; Cancel the deploy job if the run time exceeds the timeout value.
- **Compare changes against** &mdash; By default, it’s set to **No deferral**. Select either **Environment** or **This Job** to let dbt Cloud know what it should compare the changes against.

:::info
Older versions of dbt Cloud only allow you to defer to a specific job instead of an environment. Deferral to a job compares state against the project code that was run in the deferred job's last successful run. While deferral to an environment is more efficient as dbt Cloud will compare against the project representation (which is stored in the `manifest.json`) of the last successful deploy job run that executed in the deferred environment. By considering _all_ deploy jobs that run in the deferred environment, dbt Cloud will get a more accurate, latest project representation state.
:::

- **dbt Version** &mdash; By default, it’s set to inherit the [dbt version](/docs/dbt-versions/core) from the environment. dbt Labs strongly recommends that you don't change the default setting. This option to change the version at the job level is useful only when you upgrade a project to the next dbt version; otherwise, mismatched versions between the environment and job can lead to confusing behavior.
- **dbt version** &mdash; By default, it’s set to inherit the [dbt version](/docs/dbt-versions/core) from the environment. dbt Labs strongly recommends that you don't change the default setting. This option to change the version at the job level is useful only when you upgrade a project to the next dbt version; otherwise, mismatched versions between the environment and job can lead to confusing behavior.
- **Threads** &mdash; By default, it’s set to 4 [threads](/docs/core/connect-data-platform/connection-profiles#understanding-threads). Increase the thread count to increase model execution concurrency.

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/deploy-job-adv-settings.png" width="90%" title="Example of Advanced Settings on Deploy Job page"/>
<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/deploy-job-adv-settings.png" width="90%" title="Example of Advanced Settings on the Deploy Job page"/>

### Schedule days

To set your job's schedule, use the **Schedule Days** option to choose specific days of the week, and select customized hours or intervals.
To set your job's schedule, use the **Run on schedule** option to choose specific days of the week, and select customized hours or intervals.

Under **Timing**, you can either use customizable hours for jobs that need to run frequently throughout the day or exact intervals for jobs that need to run at specific times:

- **Every n hours** &mdash; Use this option to set how often your job runs, in hours. Enter a number between 1 and 23 to represent the interval between job runs. For example, if you set it to "every 2 hours", the job will run every 2 hours from midnight UTC. This option is useful if you need to run jobs multiple times per day at regular intervals.
- **Exact intervals** &mdash; Use this option to set how often your job runs, in hours. Enter a number between 1 and 23 to represent the interval between job runs. For example, if you set it to **Every 2 hours**, the job will run every 2 hours from midnight UTC. This option is useful if you need to run jobs multiple times per day at regular intervals.

- **At exact intervals** &mdash; Use this option to set specific times when your job should run. You can enter a comma-separated list of hours (in UTC) when you want the job to run. For example, if you set it to `0,12,23,` the job will run at midnight, noon, and 11 PM UTC. This option is useful if you want your jobs to run at specific times of day and don't need them to run more frequently than once a day.
- **Hours of the day** &mdash; Use this option to set specific times when your job should run. You can enter a comma-separated list of hours (in UTC) when you want the job to run. For example, if you set it to `0,12,23,` the job will run at midnight, noon, and 11 PM UTC. This option is useful if you want your jobs to run at specific times of day and don't need them to run more frequently than once a day.

:::info

Expand All @@ -75,12 +81,9 @@ dbt Cloud uses [Coordinated Universal Time](https://en.wikipedia.org/wiki/Coordi

:::

### Custom cron schedule

To fully customize the scheduling of your job, choose the **Custom cron schedule** option and use the cron syntax. With this syntax, you can specify the minute, hour, day of the month, month, and day of the week, allowing you to set up complex schedules like running a job on the first Monday of each month.
### Cron schedule


<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/job-schedule.png" title="Schedule your dbt job"/>
To fully customize the scheduling of your job, choose the **Cron schedule** option and use cron syntax. With this syntax, you can specify the minute, hour, day of the month, month, and day of the week, allowing you to set up complex schedules like running a job on the first Monday of each month.

Use tools such as [crontab.guru](https://crontab.guru/) to generate the correct cron syntax. This tool allows you to input cron snippets and returns their plain English translations.

Expand All @@ -101,6 +104,15 @@ Here are examples of cron job schedules. The dbt Cloud job scheduler supports us
- `0 7 L * 5`: At 07:00 AM, on the last day of the month, and on Friday.
- `30 14 L * *`: At 02:30 PM, on the last day of the month.

### Trigger on job completion <Lifecycle status="team,enterprise" />

To _chain_ deploy jobs together, enable the **Run when another job finishes** option and specify the upstream (parent) job that, when it completes, will trigger your job. You can also use the [Create Job API](/dbt-cloud/api-v2#/operations/Create%20Job) to do this.

You can set up a configuration where an upstream job triggers multiple downstream (child) jobs and jobs in other projects. You must have proper [permissions](/docs/cloud/manage-access/enterprise-permissions#project-role-permissions) to the project and job to configure the trigger.

For jobs that are triggered to run by another job, a link to the upstream job run is available from your [job's run details](/docs/deploy/run-visibility#job-run-details).


## Related docs

- [Artifacts](/docs/deploy/artifacts)
Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/deploy/job-scheduler.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ Familiarize yourself with these useful terms to help you understand how the job

## Scheduler queue

The scheduler queues a deployment job to be processed when it's triggered to run because of a [set schedule](#create-and-schedule-jobs), an API call, or manual action.
The scheduler queues a deployment job to be processed when it's triggered to run by a [set schedule](/docs/deploy/deploy-jobs#schedule-days), [a job completed](/docs/deploy/deploy-jobs#trigger-on-job-completion), an API call, or manual action.

Before the job starts executing, the scheduler checks these conditions to determine if the run can start executing:

Expand Down
Loading

0 comments on commit 703705a

Please sign in to comment.