Skip to content

Commit

Permalink
add DBT_EXPORT_SAVED_QUERIES (#5024)
Browse files Browse the repository at this point in the history
- Rename DBT_INCLUDE_SAVED_QUERY to DBT_EXPORT_SAVED_QUERIES
- switch out "materialize" for "write in various pages such as
'exports', 'saved queries', and the 'exports release note'

cc @Jstein77 @jtcohen6 and @matthewshaver 

Resolves #5018
  • Loading branch information
mirnawong1 authored Mar 14, 2024
2 parents 7319aee + 8994018 commit 6e48310
Show file tree
Hide file tree
Showing 6 changed files with 51 additions and 23 deletions.
4 changes: 2 additions & 2 deletions website/docs/docs/build/saved-queries.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ Saved queries are distinct from [exports](/docs/use-dbt-semantic-layer/exports),
| Feature | Exports | <div style={{width:'250px, text-align: center'}}>Saved queries</div> |
| ----------- | ----------- | ---------------- |
| **Availability** | Available on dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) plans on dbt versions 1.7 or newer.| Available in both dbt Core and dbt Cloud. |
| **Purpose** | To materialize saved queries in your data platform and expose metrics and dimensions as a view or table. | To define and manage common Semantic Layer queries in YAML, which includes metrics and dimensions. |
| **Usage** | Automatically runs saved queries and materializes them within your data platform. Exports count towards [queried metrics](/docs/cloud/billing#what-counts-as-a-queried-metric) usage. <br /><br />**Example**: Creating a weekly aggregated table for active user metrics, automatically updated and stored in the data platform. | Used for organizing and reusing common MetricFlow queries within dbt projects.<br /><br /><br />**Example**: Group related metrics together for better organization, and include commonly uses dimensions and filters. | For materializing query results in the data platform. |
| **Purpose** | To write saved queries in your data platform and expose metrics and dimensions as a view or table. | To define and manage common Semantic Layer queries in YAML, which includes metrics and dimensions. |
| **Usage** | Automatically runs saved queries and writes them within your data platform. Exports count towards [queried metrics](/docs/cloud/billing#what-counts-as-a-queried-metric) usage. <br /><br />**Example**: Creating a weekly aggregated table for active user metrics, automatically updated and stored in the data platform. | Used for organizing and reusing common MetricFlow queries within dbt projects.<br /><br /><br />**Example**: Group related metrics together for better organization, and include commonly uses dimensions and filters. | For materializing query results in the data platform. |
| **Integration** | Must have the dbt Semantic Layer configured in your dbt project.<br /><br />Tightly integrated with the [MetricFlow Server](/docs/use-dbt-semantic-layer/sl-architecture#components) and dbt Cloud's job scheduler. | Integrated into the dbt <Term id="dag" /> and managed alongside other dbt nodes. |
| **Configuration** | Defined within the `saved_queries` configuration. Set up within the dbt Cloud environment and job scheduler settings. | Defined in YAML format within dbt project files. |

Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
---
title: "New: Use exports to materialize saved queries"
description: "February 2024: Use the exports feature to materialize and schedule saved queries with dbt Cloud, and integrate with additional tools."
sidebar_label: "New: Use exports to materialize saved queries"
title: "New: Use exports to write saved queries"
description: "February 2024: Use the exports feature to write and schedule saved queries with dbt Cloud, and integrate with additional tools."
sidebar_label: "New: Use exports to write saved queries"
sidebar_position: 09
tags: [Feb-2024]
date: 2024-02-07
---

You can now use the [exports](/docs/use-dbt-semantic-layer/exports) feature with [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl), allowing you to query reliable metrics and fast data reporting. Exports enhance the saved queries feature, allowing you to materialize commonly used queries directly within your data platform using dbt Cloud's job scheduler.
You can now use the [exports](/docs/use-dbt-semantic-layer/exports) feature with [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl), allowing you to query reliable metrics and fast data reporting. Exports enhance the saved queries feature, allowing you to write commonly used queries directly within your data platform using dbt Cloud's job scheduler.

By exposing tables of metrics and dimensions, exports enable you to integrate with additional tools that don't natively connect with the dbt Semantic Layer, such as PowerBI.

Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/deploy/deployment-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ dbt Cloud offers the easiest and most reliable way to run your dbt project in pr
- Identify the root cause of failures in deployment environments
- Maintain high-quality code and data in production
- Gain visibility into the health of deployment jobs, models, and tests
- Uses [exports](/docs/use-dbt-semantic-layer/exports) to materialize [saved queries](/docs/build/saved-queries) in your data platform for reliable and fast metric reporting
- Uses [exports](/docs/use-dbt-semantic-layer/exports) to write [saved queries](/docs/build/saved-queries) in your data platform for reliable and fast metric reporting

Before continuing, make sure you understand dbt's approach to [deployment environments](/docs/deploy/deploy-environments).

Expand Down
4 changes: 2 additions & 2 deletions website/docs/docs/use-dbt-semantic-layer/dbt-sl.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,8 @@ plan="dbt Cloud Team or Enterprise"
icon="dbt-bit"/>

<Card
title="Materialize queries with exports"
body="Use exports to materialize commonly used queries directly within your data platform, on a schedule."
title="Write queries with exports"
body="Use exports to write commonly used queries directly within your data platform, on a schedule."
link="/docs/use-dbt-semantic-layer/exports"
icon="dbt-bit"/>

Expand Down
56 changes: 42 additions & 14 deletions website/docs/docs/use-dbt-semantic-layer/exports.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,14 @@
---
title: "Materialize queries with exports"
description: "Use exports to materialize tables to the data platform on a schedule."
sidebar_label: "Materialize with exports"
title: "Write queries with exports"
description: "Use exports to write tables to the data platform on a schedule."
sidebar_label: "Write commonly used queries"
---

# Materialize queries with exports <Lifecycle status='new' />

The exports feature in the dbt Semantic Layer enhances the [saved queries](/docs/build/saved-queries) by allowing you to materialize commonly used queries directly within your data platform.
The exports feature in the dbt Semantic Layer enhances the [saved queries](/docs/build/saved-queries) by allowing you to write commonly used queries directly within your data platform.

While saved queries are a way to save and reuse commonly used queries in MetricFlow, exports take this functionality a step further by:

- Enabling you to materialize these queries within your data platform using dbt Cloud.
- Enabling you to write these queries within your data platform using dbt Cloud.
- Proving an integration path for tools that don't natively support the dbt Semantic Layer by exposing tables of metrics and dimensions.

Essentially, exports are like any other table in your data platform. They enable you to query metric definitions through any SQL interface or connect to downstream tools without a first-class Semantic Layer integration. Refer to [Available integrations](/docs/use-dbt-semantic-layer/avail-sl-integrations) for more information.
Expand All @@ -29,14 +27,14 @@ Essentially, exports are like any other table in your data platform. They enable
| Feature | Exports | <div style={{width:'250px, text-align: center'}}>Saved queries</div> |
| ----------- | ----------- | ---------------- |
| **Availability** | Available on dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) plans with dbt versions 1.7 or newer.| Available in both dbt Core and dbt Cloud. |
| **Purpose** | To materialize saved queries in your data platform and expose metrics and dimensions as a view or table. | To define and manage common Semantic Layer queries in YAML, including metrics and dimensions. |
| **Usage** | Automatically runs saved queries and materializes them within your data platform. Exports count towards [queried metrics](/docs/cloud/billing#what-counts-as-a-queried-metric) usage. <br /><br />Example: Create a weekly aggregated table for active user metrics, automatically updated and stored in the data platform. | Used for organizing and reusing common MetricFlow queries within dbt projects.<br /><br /><br />Example: Group related metrics together for better organization, and include commonly used dimensions and filters. | For materializing query results in the data platform. |
| **Purpose** | To write saved queries in your data platform and expose metrics and dimensions as a view or table. | To define and manage common Semantic Layer queries in YAML, including metrics and dimensions. |
| **Usage** | Automatically runs saved queries and writes them within your data platform. Exports count towards [queried metrics](/docs/cloud/billing#what-counts-as-a-queried-metric) usage. <br /><br />Example: Create a weekly aggregated table for active user metrics, automatically updated and stored in the data platform. | Used for organizing and reusing common MetricFlow queries within dbt projects.<br /><br /><br />Example: Group related metrics together for better organization, and include commonly used dimensions and filters. | For materializing query results in the data platform. |
| **Integration** | Must have the dbt Semantic Layer configured in your dbt project.<br /><br />Tightly integrated with the [MetricFlow Server](/docs/use-dbt-semantic-layer/sl-architecture#components) and dbt Cloud's job scheduler. | Integrated into the dbt <Term id="dag" /> and managed alongside other dbt nodes. |
| **Configuration** | Defined within the `saved_queries` configuration. Set up within the dbt Cloud environment and job scheduler settings. | Defined in YAML format within dbt project files. |

## Define exports

Exports are an additional configuration added to a saved query. They define how to materialize a saved query, along with the schema and table name.
Exports are an additional configuration added to a saved query. They define how to write a saved query, along with the schema and table name.

You can define `exports` in YAML format as a key within the `saved_queries` configuration and in the same file as your metric definitions.

Expand Down Expand Up @@ -92,7 +90,7 @@ You can run an export in your development environment using your development cre
dbt sl export
```

You can also run any export defined for the saved query and materialize the table or view in your development environment. Refer to the following command example and output:
You can also run any export defined for the saved query and write the table or view in your development environment. Refer to the following command example and output:

**Example**
```bash
Expand Down Expand Up @@ -121,8 +119,8 @@ dbt sl export --saved-query sq_name --select export_1,export2
The `--select` flag is mainly used to include or exclude specific exports. If you need to change these settings, you can use the following flags to override export configurations:

- `--export-as` &mdash; Defines the materialization type (table or view) for the export. This creates a new export with its own settings and is useful for testing in development.
- `--schema` &mdash; Specifies the schema to use for the materialized table or view.
- `--alias` &mdash; Assigns a custom alias to the materialized table or view. This overrides the default export name.
- `--schema` &mdash; Specifies the schema to use for the written table or view.
- `--alias` &mdash; Assigns a custom alias to the written table or view. This overrides the default export name.

Be careful. The `--select` flag _can't_ be used with `alias` or `schema`.

Expand All @@ -141,6 +139,8 @@ To enable exports in production using the dbt Cloud job scheduler:
1. [Create and execute exports](#create-and-execute-exports)

#### Set environment variable
<!-- for version 1.7 -->
<VersionBlock firstVersion lastVersion="1.7">

1. Click **Deploy** in the top navigation bar and choose **Environments**.
2. Select **Environment variables**.
Expand All @@ -152,12 +152,40 @@ If exports aren't needed, you can set the value(s) to `FALSE` (`DBT_INCLUDE_SAVE

<Lightbox src="/img/docs/dbt-cloud/semantic-layer/deploy_exports.jpg" width="90%" title="Add an environment variable to run exports in your production run." />

</VersionBlock>

<!-- for keep on latest version -->
<VersionBlock firstVersion="1.8">

1. Click **Deploy** in the top navigation bar and choose **Environments**.
2. Select **Environment variables**.
3. [Set the environment variable](/docs/build/environment-variables#setting-and-overriding-environment-variables) key to `DBT_EXPORT_SAVED_QUERIES` and the environment variable's value to `TRUE` (`DBT_EXPORT_SAVED_QUERIES=TRUE`).

Doing this ensures saved queries and exports are included in your dbt build job. For example, running `dbt build sq_name` runs the equivalent of `dbt sl export --saved-query sq_name` in the dbt Cloud Job scheduler.

If exports aren't needed, you can set the value(s) to `FALSE` (`DBT_EXPORT_SAVED_QUERIES=FALSE`).

<Lightbox src="/img/docs/dbt-cloud/semantic-layer/env-var-dbt-exports.jpg" width="90%" title="Add an environment variable to run exports in your production run." />

</VersionBlock>

When you run a build job, any saved queries downstream of the dbt models in that job will also run. To make sure your export data is up-to-date, run the export as a downstream step (after the model).

#### Create and execute exports
<VersionBlock firstVersion lastVersion="1.7">

1. Create a [deploy job](/docs/deploy/deploy-jobs) and ensure the `DBT_INCLUDE_SAVED_QUERY=TRUE` environment variable is set, as described in [Set environment variable](#set-environment-variable).
- This enables you to run any export that needs to be refreshed after a model is built.

1. Create a [deploy job](/docs/deploy/deploy-jobs) and ensure the `DBT_INCLUDE_SAVED_QUERY=TRUE` environment variable is set, as described in [Set environment variable](#set-environment-variable).
</VersionBlock>

<VersionBlock firstVersion="1.8">

1. Create a [deploy job](/docs/deploy/deploy-jobs) and ensure the `DBT_EXPORT_SAVED_QUERIES=TRUE` environment variable is set, as described in [Set environment variable](#set-environment-variable).
- This enables you to run any export that needs to be refreshed after a model is built.

</VersionBlock>

2. After dbt finishes building the models, the MetricFlow Server processes the exports, compiles the necessary SQL, and executes this SQL against your data platform.
3. Review the exports' execution details in the jobs logs and confirm the export was run successfully. This helps troubleshoot and to ensure accuracy. Since saved queries are integrated into the dbt DAG, all outputs related to exports are available in the job logs.
4. Your data is now available in the data platform for querying.
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 6e48310

Please sign in to comment.