Skip to content

Commit

Permalink
Merge branch 'current' into dbeatty10-patch-1
Browse files Browse the repository at this point in the history
  • Loading branch information
dbeatty10 authored Mar 14, 2024
2 parents f3a81f7 + 05fc390 commit c141981
Show file tree
Hide file tree
Showing 8 changed files with 136 additions and 23 deletions.
42 changes: 42 additions & 0 deletions website/docs/docs/build/metrics-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,52 @@ Once you've created your semantic models, it's time to start adding metrics! Met

The keys for metrics definitions are:

<!-- for v1.8 and higher -->

<VersionBlock firstVersion="1.8">

| Parameter | Description | Type |
| --------- | ----------- | ---- |
| `name` | Provide the reference name for the metric. This name must be unique amongst all metrics. | Required |
| `description` | Describe your metric. | Optional |
| `type` | Define the type of metric, which can be `simple`, `ratio`, `cumulative`, or `derived`. | Required |
| `type_params` | Additional parameters used to configure metrics. `type_params` are different for each metric type. | Required |
| `config` | Provide the specific configurations for your metric. | Optional |
| `config:meta` | Use the [`meta` config](/reference/resource-configs/meta) to set metadata for a resource. | Optional |
| `label` | The display name for your metric. This value will be shown in downstream tools. | Required |
| `filter` | You can optionally add a filter string to any metric type, applying filters to dimensions, entities, or time dimensions during metric computation. Consider it as your WHERE clause. | Optional |

Here's a complete example of the metrics spec configuration:

```yaml
metrics:
- name: metric name ## Required
description: same as always ## Optional
type: the type of the metric ## Required
type_params: ## Required
- specific properties for the metric type
config: here for `enabled` ## Optional
meta:
my_meta_config: 'config' ## Optional
label: The display name for your metric. This value will be shown in downstream tools. ## Required
filter: | ## Optional
{{ Dimension('entity__name') }} > 0 and {{ Dimension(' entity__another_name') }} is not
null
```
</VersionBlock>
<!-- for v1.7 and lower -->
<VersionBlock lastVersion="1.7">
| Parameter | Description | Type |
| --------- | ----------- | ---- |
| `name` | Provide the reference name for the metric. This name must be unique amongst all metrics. | Required |
| `description` | Describe your metric. | Optional |
| `type` | Define the type of metric, which can be `simple`, `ratio`, `cumulative`, or `derived`. | Required |
| `type_params` | Additional parameters used to configure metrics. `type_params` are different for each metric type. | Required |
| `config` | Provide the specific configurations for your metric. | Optional |
| `meta` | Use the [`meta` config](/reference/resource-configs/meta) to set metadata for a resource. | Optional |
| `label` | The display name for your metric. This value will be shown in downstream tools. | Required |
| `filter` | You can optionally add a filter string to any metric type, applying filters to dimensions, entities, or time dimensions during metric computation. Consider it as your WHERE clause. | Optional |

Expand All @@ -31,11 +70,14 @@ metrics:
type_params: ## Required
- specific properties for the metric type
config: here for `enabled` ## Optional
meta:
my_meta_direct: 'direct' ## Optional
label: The display name for your metric. This value will be shown in downstream tools. ## Required
filter: | ## Optional
{{ Dimension('entity__name') }} > 0 and {{ Dimension(' entity__another_name') }} is not
null
```
</VersionBlock>
This page explains the different supported metric types you can add to your dbt project.
Expand Down
4 changes: 2 additions & 2 deletions website/docs/docs/build/saved-queries.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ Saved queries are distinct from [exports](/docs/use-dbt-semantic-layer/exports),
| Feature | Exports | <div style={{width:'250px, text-align: center'}}>Saved queries</div> |
| ----------- | ----------- | ---------------- |
| **Availability** | Available on dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) plans on dbt versions 1.7 or newer.| Available in both dbt Core and dbt Cloud. |
| **Purpose** | To materialize saved queries in your data platform and expose metrics and dimensions as a view or table. | To define and manage common Semantic Layer queries in YAML, which includes metrics and dimensions. |
| **Usage** | Automatically runs saved queries and materializes them within your data platform. Exports count towards [queried metrics](/docs/cloud/billing#what-counts-as-a-queried-metric) usage. <br /><br />**Example**: Creating a weekly aggregated table for active user metrics, automatically updated and stored in the data platform. | Used for organizing and reusing common MetricFlow queries within dbt projects.<br /><br /><br />**Example**: Group related metrics together for better organization, and include commonly uses dimensions and filters. | For materializing query results in the data platform. |
| **Purpose** | To write saved queries in your data platform and expose metrics and dimensions as a view or table. | To define and manage common Semantic Layer queries in YAML, which includes metrics and dimensions. |
| **Usage** | Automatically runs saved queries and writes them within your data platform. Exports count towards [queried metrics](/docs/cloud/billing#what-counts-as-a-queried-metric) usage. <br /><br />**Example**: Creating a weekly aggregated table for active user metrics, automatically updated and stored in the data platform. | Used for organizing and reusing common MetricFlow queries within dbt projects.<br /><br /><br />**Example**: Group related metrics together for better organization, and include commonly uses dimensions and filters. | For materializing query results in the data platform. |
| **Integration** | Must have the dbt Semantic Layer configured in your dbt project.<br /><br />Tightly integrated with the [MetricFlow Server](/docs/use-dbt-semantic-layer/sl-architecture#components) and dbt Cloud's job scheduler. | Integrated into the dbt <Term id="dag" /> and managed alongside other dbt nodes. |
| **Configuration** | Defined within the `saved_queries` configuration. Set up within the dbt Cloud environment and job scheduler settings. | Defined in YAML format within dbt project files. |

Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
---
title: "New: Use exports to materialize saved queries"
description: "February 2024: Use the exports feature to materialize and schedule saved queries with dbt Cloud, and integrate with additional tools."
sidebar_label: "New: Use exports to materialize saved queries"
title: "New: Use exports to write saved queries"
description: "February 2024: Use the exports feature to write and schedule saved queries with dbt Cloud, and integrate with additional tools."
sidebar_label: "New: Use exports to write saved queries"
sidebar_position: 09
tags: [Feb-2024]
date: 2024-02-07
---

You can now use the [exports](/docs/use-dbt-semantic-layer/exports) feature with [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl), allowing you to query reliable metrics and fast data reporting. Exports enhance the saved queries feature, allowing you to materialize commonly used queries directly within your data platform using dbt Cloud's job scheduler.
You can now use the [exports](/docs/use-dbt-semantic-layer/exports) feature with [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl), allowing you to query reliable metrics and fast data reporting. Exports enhance the saved queries feature, allowing you to write commonly used queries directly within your data platform using dbt Cloud's job scheduler.

By exposing tables of metrics and dimensions, exports enable you to integrate with additional tools that don't natively connect with the dbt Semantic Layer, such as PowerBI.

Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/deploy/deployment-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ dbt Cloud offers the easiest and most reliable way to run your dbt project in pr
- Identify the root cause of failures in deployment environments
- Maintain high-quality code and data in production
- Gain visibility into the health of deployment jobs, models, and tests
- Uses [exports](/docs/use-dbt-semantic-layer/exports) to materialize [saved queries](/docs/build/saved-queries) in your data platform for reliable and fast metric reporting
- Uses [exports](/docs/use-dbt-semantic-layer/exports) to write [saved queries](/docs/build/saved-queries) in your data platform for reliable and fast metric reporting

Before continuing, make sure you understand dbt's approach to [deployment environments](/docs/deploy/deploy-environments).

Expand Down
4 changes: 2 additions & 2 deletions website/docs/docs/use-dbt-semantic-layer/dbt-sl.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,8 @@ plan="dbt Cloud Team or Enterprise"
icon="dbt-bit"/>

<Card
title="Materialize queries with exports"
body="Use exports to materialize commonly used queries directly within your data platform, on a schedule."
title="Write queries with exports"
body="Use exports to write commonly used queries directly within your data platform, on a schedule."
link="/docs/use-dbt-semantic-layer/exports"
icon="dbt-bit"/>

Expand Down
56 changes: 42 additions & 14 deletions website/docs/docs/use-dbt-semantic-layer/exports.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,14 @@
---
title: "Materialize queries with exports"
description: "Use exports to materialize tables to the data platform on a schedule."
sidebar_label: "Materialize with exports"
title: "Write queries with exports"
description: "Use exports to write tables to the data platform on a schedule."
sidebar_label: "Write commonly used queries"
---

# Materialize queries with exports <Lifecycle status='new' />

The exports feature in the dbt Semantic Layer enhances the [saved queries](/docs/build/saved-queries) by allowing you to materialize commonly used queries directly within your data platform.
The exports feature in the dbt Semantic Layer enhances the [saved queries](/docs/build/saved-queries) by allowing you to write commonly used queries directly within your data platform.

While saved queries are a way to save and reuse commonly used queries in MetricFlow, exports take this functionality a step further by:

- Enabling you to materialize these queries within your data platform using dbt Cloud.
- Enabling you to write these queries within your data platform using dbt Cloud.
- Proving an integration path for tools that don't natively support the dbt Semantic Layer by exposing tables of metrics and dimensions.

Essentially, exports are like any other table in your data platform. They enable you to query metric definitions through any SQL interface or connect to downstream tools without a first-class Semantic Layer integration. Refer to [Available integrations](/docs/use-dbt-semantic-layer/avail-sl-integrations) for more information.
Expand All @@ -29,14 +27,14 @@ Essentially, exports are like any other table in your data platform. They enable
| Feature | Exports | <div style={{width:'250px, text-align: center'}}>Saved queries</div> |
| ----------- | ----------- | ---------------- |
| **Availability** | Available on dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) plans with dbt versions 1.7 or newer.| Available in both dbt Core and dbt Cloud. |
| **Purpose** | To materialize saved queries in your data platform and expose metrics and dimensions as a view or table. | To define and manage common Semantic Layer queries in YAML, including metrics and dimensions. |
| **Usage** | Automatically runs saved queries and materializes them within your data platform. Exports count towards [queried metrics](/docs/cloud/billing#what-counts-as-a-queried-metric) usage. <br /><br />Example: Create a weekly aggregated table for active user metrics, automatically updated and stored in the data platform. | Used for organizing and reusing common MetricFlow queries within dbt projects.<br /><br /><br />Example: Group related metrics together for better organization, and include commonly used dimensions and filters. | For materializing query results in the data platform. |
| **Purpose** | To write saved queries in your data platform and expose metrics and dimensions as a view or table. | To define and manage common Semantic Layer queries in YAML, including metrics and dimensions. |
| **Usage** | Automatically runs saved queries and writes them within your data platform. Exports count towards [queried metrics](/docs/cloud/billing#what-counts-as-a-queried-metric) usage. <br /><br />Example: Create a weekly aggregated table for active user metrics, automatically updated and stored in the data platform. | Used for organizing and reusing common MetricFlow queries within dbt projects.<br /><br /><br />Example: Group related metrics together for better organization, and include commonly used dimensions and filters. | For materializing query results in the data platform. |
| **Integration** | Must have the dbt Semantic Layer configured in your dbt project.<br /><br />Tightly integrated with the [MetricFlow Server](/docs/use-dbt-semantic-layer/sl-architecture#components) and dbt Cloud's job scheduler. | Integrated into the dbt <Term id="dag" /> and managed alongside other dbt nodes. |
| **Configuration** | Defined within the `saved_queries` configuration. Set up within the dbt Cloud environment and job scheduler settings. | Defined in YAML format within dbt project files. |

## Define exports

Exports are an additional configuration added to a saved query. They define how to materialize a saved query, along with the schema and table name.
Exports are an additional configuration added to a saved query. They define how to write a saved query, along with the schema and table name.

You can define `exports` in YAML format as a key within the `saved_queries` configuration and in the same file as your metric definitions.

Expand Down Expand Up @@ -92,7 +90,7 @@ You can run an export in your development environment using your development cre
dbt sl export
```

You can also run any export defined for the saved query and materialize the table or view in your development environment. Refer to the following command example and output:
You can also run any export defined for the saved query and write the table or view in your development environment. Refer to the following command example and output:

**Example**
```bash
Expand Down Expand Up @@ -121,8 +119,8 @@ dbt sl export --saved-query sq_name --select export_1,export2
The `--select` flag is mainly used to include or exclude specific exports. If you need to change these settings, you can use the following flags to override export configurations:

- `--export-as` &mdash; Defines the materialization type (table or view) for the export. This creates a new export with its own settings and is useful for testing in development.
- `--schema` &mdash; Specifies the schema to use for the materialized table or view.
- `--alias` &mdash; Assigns a custom alias to the materialized table or view. This overrides the default export name.
- `--schema` &mdash; Specifies the schema to use for the written table or view.
- `--alias` &mdash; Assigns a custom alias to the written table or view. This overrides the default export name.

Be careful. The `--select` flag _can't_ be used with `alias` or `schema`.

Expand All @@ -141,6 +139,8 @@ To enable exports in production using the dbt Cloud job scheduler:
1. [Create and execute exports](#create-and-execute-exports)

#### Set environment variable
<!-- for version 1.7 -->
<VersionBlock firstVersion lastVersion="1.7">

1. Click **Deploy** in the top navigation bar and choose **Environments**.
2. Select **Environment variables**.
Expand All @@ -152,12 +152,40 @@ If exports aren't needed, you can set the value(s) to `FALSE` (`DBT_INCLUDE_SAVE

<Lightbox src="/img/docs/dbt-cloud/semantic-layer/deploy_exports.jpg" width="90%" title="Add an environment variable to run exports in your production run." />

</VersionBlock>

<!-- for keep on latest version -->
<VersionBlock firstVersion="1.8">

1. Click **Deploy** in the top navigation bar and choose **Environments**.
2. Select **Environment variables**.
3. [Set the environment variable](/docs/build/environment-variables#setting-and-overriding-environment-variables) key to `DBT_EXPORT_SAVED_QUERIES` and the environment variable's value to `TRUE` (`DBT_EXPORT_SAVED_QUERIES=TRUE`).

Doing this ensures saved queries and exports are included in your dbt build job. For example, running `dbt build sq_name` runs the equivalent of `dbt sl export --saved-query sq_name` in the dbt Cloud Job scheduler.

If exports aren't needed, you can set the value(s) to `FALSE` (`DBT_EXPORT_SAVED_QUERIES=FALSE`).

<Lightbox src="/img/docs/dbt-cloud/semantic-layer/env-var-dbt-exports.jpg" width="90%" title="Add an environment variable to run exports in your production run." />

</VersionBlock>

When you run a build job, any saved queries downstream of the dbt models in that job will also run. To make sure your export data is up-to-date, run the export as a downstream step (after the model).

#### Create and execute exports
<VersionBlock firstVersion lastVersion="1.7">

1. Create a [deploy job](/docs/deploy/deploy-jobs) and ensure the `DBT_INCLUDE_SAVED_QUERY=TRUE` environment variable is set, as described in [Set environment variable](#set-environment-variable).
- This enables you to run any export that needs to be refreshed after a model is built.

1. Create a [deploy job](/docs/deploy/deploy-jobs) and ensure the `DBT_INCLUDE_SAVED_QUERY=TRUE` environment variable is set, as described in [Set environment variable](#set-environment-variable).
</VersionBlock>

<VersionBlock firstVersion="1.8">

1. Create a [deploy job](/docs/deploy/deploy-jobs) and ensure the `DBT_EXPORT_SAVED_QUERIES=TRUE` environment variable is set, as described in [Set environment variable](#set-environment-variable).
- This enables you to run any export that needs to be refreshed after a model is built.

</VersionBlock>

2. After dbt finishes building the models, the MetricFlow Server processes the exports, compiles the necessary SQL, and executes this SQL against your data platform.
3. Review the exports' execution details in the jobs logs and confirm the export was run successfully. This helps troubleshoot and to ensure accuracy. Since saved queries are integrated into the dbt DAG, all outputs related to exports are available in the job logs.
4. Your data is now available in the data platform for querying.
Expand Down
43 changes: 43 additions & 0 deletions website/docs/reference/resource-configs/meta.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ default_value: {}
{ label: 'Macros', value: 'macros', },
{ label: 'Exposures', value: 'exposures', },
{ label: 'Semantic Models', value: 'semantic models', },
{ label: 'Metrics', value: 'metrics', },
]
}>
<TabItem value="models">
Expand Down Expand Up @@ -202,6 +203,48 @@ The `meta` config can also be defined under the `semantic-models` config block i

</TabItem>

<TabItem value="metrics">

<VersionBlock lastVersion="1.7">

<File name='models/metrics.yml'>

```yml
metrics:
- name: number_of_people
label: "Number of people"
description: Total count of people
type: simple
type_params:
measure: people
meta:
my_meta_direct: 'direct'
```
</File>
</VersionBlock>
<VersionBlock firstVersion="1.8">
<File name='models/metrics.yml'>
```yml
metrics:
- name: number_of_people
label: "Number of people"
description: Total count of people
type: simple
type_params:
measure: people
config:
meta:
my_meta_config: 'config_value'
```
</File>
</VersionBlock>
</TabItem>
</Tabs>
## Definition
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit c141981

Please sign in to comment.