From eb93839faec4ef5fad5fa113d0f38d87e635fbb3 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 6 Nov 2024 11:54:57 +0000 Subject: [PATCH 01/30] add tags to exports --- website/docs/docs/build/saved-queries.md | 2 + .../docs/docs/dbt-versions/release-notes.md | 8 ++++ .../docs/use-dbt-semantic-layer/exports.md | 10 +++-- .../docs/reference/resource-configs/tags.md | 37 ++++++++++++++++++- 4 files changed, 53 insertions(+), 4 deletions(-) diff --git a/website/docs/docs/build/saved-queries.md b/website/docs/docs/build/saved-queries.md index ed56d13dcc9..d1fe785f483 100644 --- a/website/docs/docs/build/saved-queries.md +++ b/website/docs/docs/build/saved-queries.md @@ -58,6 +58,7 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `exports::config::export_as` | String | Required | Specifies the type of export: table, view, or upcoming cache options. Nested within `exports` and `config`. | | `exports::config::schema` | String | Optional | Schema for creating the table or view, not applicable for caching. Nested within `exports` and `config`. | | `exports::config::alias` | String | Optional | Table alias used to write to the table or view. This option can't be used for caching. Nested within `exports` and `config`. | +| `exports::config::tag` | String | Optional | Apply a [tag](/reference/resource-configs/tags) (or list of tags) to a resource. Tags help organize and filter resources in dbt. Nested within `exports` and `config`. | @@ -99,6 +100,7 @@ saved_queries: alias: my_export_alias export_as: table schema: my_export_schema_name + [tags](/reference/resource-configs/tags): ['my_tag'] ``` diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index c3f0bbfbe06..54326474e7c 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -24,6 +24,14 @@ Release notes are grouped by month for both multi-tenant and virtual private clo - Improved handling of queries when multiple tables are selected in a data source. - Fixed a bug when an IN filter contained a lot of values. - Better error messaging for queries that can't be parsed correctly. +- **New**: Exports now support [tags](/docs/reference/resource-configs/tags) in dbt Cloud. Tags allow you to categorize your resources and filter them in dbt Cloud. You can add tags to your [exports](/docs/build/saved-queries#configure-exports) in the `semantic_model.yml` file or `dbt_project.yml` file. For example: + + ```yml + exports: + - name: export_name + tags: ['export_tag'] + ... + ``` ## October 2024 diff --git a/website/docs/docs/use-dbt-semantic-layer/exports.md b/website/docs/docs/use-dbt-semantic-layer/exports.md index 5d6e4c0d996..af8e0dce1cd 100644 --- a/website/docs/docs/use-dbt-semantic-layer/exports.md +++ b/website/docs/docs/use-dbt-semantic-layer/exports.md @@ -29,16 +29,20 @@ The following section explains the main benefits of using exports, including: - [Easier changes](#easier-changes) - [Caching](#caching) -#### DRY representation + Currently, creating tables often involves generating tens, hundreds, or even thousands of tables that denormalize data into summary or metric mart tables. The main benefit of exports is creating a "Don't Repeat Yourself (DRY)" representation of the logic to construct each metric, dimension, join, filter, and so on. This allows you to reuse those components for long-term scalability, even if you're replacing manually written SQL models with references to the metrics or dimensions in saved queries. + -#### Easier changes + Exports ensure that changes to metrics and dimensions are made in one place and then cascade to those various destinations seamlessly. This prevents the problem of needing to update a metric across every model that references that same concept. + + + -#### Caching Use exports to pre-populate the cache, so that you're pre-computing what you need to serve users through the dynamic Semantic Layer APIs. + #### Considerations diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index f6c46f8a088..75d5c55a1af 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -30,6 +30,10 @@ seeds: [](/reference/resource-configs/resource-path): +tags: | [] +[exports](/docs/use-dbt-semantic-layer/exports): + [](/reference/resource-configs/resource-path): + +tags: | [] + ``` @@ -164,6 +168,32 @@ seeds: +### Apply tags to exports + + + +```yml +[exports](/docs/use-dbt-semantic-layer/exports): + jaffle_shop: + customer_order_metrics: + +tags: order_metrics +``` + + + + + +```yml +[exports](/docs/use-dbt-semantic-layer/exports): + jaffle_shop: + customer_order_metrics: + +tags: + - order_metrics + - hourly +``` + + + ## Usage notes ### Tags are additive @@ -178,7 +208,7 @@ Tags accumulate hierarchically. The above example would result in: ### Other resource types -Tags can also be applied to sources, exposures, and even _specific columns_ in a resource. +Tags can also be applied to [sources](/docs/build/sources), [exposures](/docs/build/exposures), [exports](/docs/use-dbt-semantic-layer/exports), and even _specific columns_ in a resource. These resources do not yet support the `config` property, so you'll need to specify the tags as a top-level key instead. @@ -206,6 +236,11 @@ sources: tests: - unique: tags: ['test_level'] + +exports: + - name: export_name + tags: ['export_tag'] + ... ``` From 744352134d1761094eb7e68a1f6abe34ba76c4af Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 6 Nov 2024 12:02:33 +0000 Subject: [PATCH 02/30] fix link --- website/docs/docs/build/saved-queries.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/saved-queries.md b/website/docs/docs/build/saved-queries.md index d1fe785f483..db978ece436 100644 --- a/website/docs/docs/build/saved-queries.md +++ b/website/docs/docs/build/saved-queries.md @@ -100,7 +100,7 @@ saved_queries: alias: my_export_alias export_as: table schema: my_export_schema_name - [tags](/reference/resource-configs/tags): ['my_tag'] + [tags](/reference/resource-configs/tags): 'my_tag' ``` From 44b92484699a3e07b41634e43f6fef16a2db84b0 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 6 Nov 2024 12:02:57 +0000 Subject: [PATCH 03/30] fix link --- website/docs/docs/dbt-versions/release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index 54326474e7c..f7faac30645 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -24,7 +24,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo - Improved handling of queries when multiple tables are selected in a data source. - Fixed a bug when an IN filter contained a lot of values. - Better error messaging for queries that can't be parsed correctly. -- **New**: Exports now support [tags](/docs/reference/resource-configs/tags) in dbt Cloud. Tags allow you to categorize your resources and filter them in dbt Cloud. You can add tags to your [exports](/docs/build/saved-queries#configure-exports) in the `semantic_model.yml` file or `dbt_project.yml` file. For example: +- **New**: Exports now support [tags](/reference/resource-configs/tags) in dbt Cloud. Tags allow you to categorize your resources and filter them in dbt Cloud. You can add tags to your [exports](/docs/build/saved-queries#configure-exports) in the `semantic_model.yml` file or `dbt_project.yml` file. For example: ```yml exports: From 3571c0f0ec44aac230cbd7680962bc8be8db163d Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 6 Nov 2024 12:43:24 +0000 Subject: [PATCH 04/30] add meta --- website/docs/docs/build/dimensions.md | 7 +++ .../docs/docs/dbt-versions/release-notes.md | 1 + .../docs/reference/resource-configs/meta.md | 52 +++++++++++++++++++ 3 files changed, 60 insertions(+) diff --git a/website/docs/docs/build/dimensions.md b/website/docs/docs/build/dimensions.md index 5026f4c45cd..eb1809ea72b 100644 --- a/website/docs/docs/build/dimensions.md +++ b/website/docs/docs/build/dimensions.md @@ -22,6 +22,7 @@ All dimensions require a `name`, `type`, and can optionally include an `expr` pa | `description` | A clear description of the dimension | Optional | | `expr` | Defines the underlying column or SQL query for a dimension. If no `expr` is specified, MetricFlow will use the column with the same name as the group. You can use the column name itself to input a SQL expression. | Optional | | `label` | A recommended string that defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | +| `meta` | Use the [`meta` field](/reference/resource-configs/meta) to set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | Refer to the following for the complete specification for dimensions: @@ -59,6 +60,8 @@ semantic_models: type_params: time_granularity: day label: "Date of transaction" # Recommend adding a label to provide more context to users consuming the data + meta: + data_owner: "Finance team" expr: ts - name: is_bulk type: categorical @@ -106,6 +109,8 @@ dimensions: - name: is_bulk_transaction type: categorical expr: case when quantity > 10 then true else false end + meta: + usage: "Filter to identify bulk transactions, like where quantity > 10." ``` ## Time @@ -136,6 +141,8 @@ dimensions: type: time label: "Date of creation" expr: ts_created # ts_created is the underlying column name from the table + meta: + notes: "Only valid for orders from 2022 onward" is_partition: True type_params: time_granularity: day diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index c3f0bbfbe06..8dd16e99bbb 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -24,6 +24,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo - Improved handling of queries when multiple tables are selected in a data source. - Fixed a bug when an IN filter contained a lot of values. - Better error messaging for queries that can't be parsed correctly. +- **New**: [Dimensions](/reference/resource-configs/meta) now support the meta config property in dbt Cloud Versionless ??? or v1.9+??. You can add metadata to your dimensions to provide additional context and information about the dimension. Refer to [meta](/reference/resource-configs/meta) for more information. ## October 2024 diff --git a/website/docs/reference/resource-configs/meta.md b/website/docs/reference/resource-configs/meta.md index 53a4f77184e..ce33feff62e 100644 --- a/website/docs/reference/resource-configs/meta.md +++ b/website/docs/reference/resource-configs/meta.md @@ -19,6 +19,7 @@ hide_table_of_contents: true { label: 'Semantic Models', value: 'semantic models', }, { label: 'Metrics', value: 'metrics', }, { label: 'Saved queries', value: 'saved queries', }, + { label: 'Dimensions', value: 'dimensions', }, ] }> @@ -250,6 +251,22 @@ saved_queries: + + + + +```yml +semantic_models: + - name: semantic_model + ... + dimensions: + - name: order_date + config: + meta: {} +``` + + + @@ -343,3 +360,38 @@ models: +### Assign owner and additional metadata to dimensions + +The following example shows how to assign a `data_owner` and additional metadata value to a dimension in a semantic model. + + + +```yml +semantic_models: + - name: semantic_model + ... + dimensions: + - name: order_date + meta: + data_owner: "Finance team" + used_in_reporting: true +``` + + + +This second example shows how to assign a `data_owner` and additional metadata value to a dimension in the `dbt_project.yml` file using the `+meta` syntax. + + + +```yml +semantic-models: + jaffle_shop: + ... + dimensions: + - name: order_date + +meta: + data_owner: "Finance team" + used_in_reporting: true +``` + + From 53c0351a95a72f6212768191b24ede084e48717b Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 6 Nov 2024 12:51:35 +0000 Subject: [PATCH 05/30] update example --- website/docs/docs/build/saved-queries.md | 2 +- .../docs/reference/resource-configs/tags.md | 28 +++++++++++++------ 2 files changed, 20 insertions(+), 10 deletions(-) diff --git a/website/docs/docs/build/saved-queries.md b/website/docs/docs/build/saved-queries.md index db978ece436..136474da3ee 100644 --- a/website/docs/docs/build/saved-queries.md +++ b/website/docs/docs/build/saved-queries.md @@ -58,7 +58,7 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `exports::config::export_as` | String | Required | Specifies the type of export: table, view, or upcoming cache options. Nested within `exports` and `config`. | | `exports::config::schema` | String | Optional | Schema for creating the table or view, not applicable for caching. Nested within `exports` and `config`. | | `exports::config::alias` | String | Optional | Table alias used to write to the table or view. This option can't be used for caching. Nested within `exports` and `config`. | -| `exports::config::tag` | String | Optional | Apply a [tag](/reference/resource-configs/tags) (or list of tags) to a resource. Tags help organize and filter resources in dbt. Nested within `exports` and `config`. | +| `exports::config::tags` | String | Optional | Apply a [tag](/reference/resource-configs/tags) (or list of tags) to a resource. Tags help organize and filter resources in dbt. Nested within `exports` and `config`. | diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index 75d5c55a1af..6018c774282 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -170,10 +170,12 @@ seeds: ### Apply tags to exports +This following example shows how to apply tags to an export in the `dbt_project.yml` file. The export is then tagged with `order_metrics`. + ```yml -[exports](/docs/use-dbt-semantic-layer/exports): +[exports](/docs/build/saved-queries-configure-exports): jaffle_shop: customer_order_metrics: +tags: order_metrics @@ -181,15 +183,23 @@ seeds: - +The second example shows how to apply tags to an export in the `semantic_model.yml` file. The export is then tagged with `order_metrics` and `hourly`. -```yml -[exports](/docs/use-dbt-semantic-layer/exports): - jaffle_shop: - customer_order_metrics: - +tags: - - order_metrics - - hourly + + +```yaml +saved_queries: + - name: order_metrics + ... + exports: + - name: hourly_order_metrics + config: + alias: my_export_alias + export_as: table + schema: my_export_schema_name + tags: + - order_metrics + - hourly ``` From 2d5a67592296524f884b5097f7b2be932fd367b6 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 6 Nov 2024 12:52:56 +0000 Subject: [PATCH 06/30] add link --- website/docs/reference/resource-configs/meta.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/reference/resource-configs/meta.md b/website/docs/reference/resource-configs/meta.md index ce33feff62e..fa0d48c299e 100644 --- a/website/docs/reference/resource-configs/meta.md +++ b/website/docs/reference/resource-configs/meta.md @@ -259,7 +259,7 @@ saved_queries: semantic_models: - name: semantic_model ... - dimensions: + [dimensions](/docs/build/dimensions): - name: order_date config: meta: {} @@ -370,7 +370,7 @@ The following example shows how to assign a `data_owner` and additional metadata semantic_models: - name: semantic_model ... - dimensions: + [dimensions](/docs/build/dimensions): - name: order_date meta: data_owner: "Finance team" @@ -387,7 +387,7 @@ This second example shows how to assign a `data_owner` and additional metadata v semantic-models: jaffle_shop: ... - dimensions: + [dimensions](/docs/build/dimensions): - name: order_date +meta: data_owner: "Finance team" From c9a08209e458134e607a19811527bcf43a475eab Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 6 Nov 2024 12:58:11 +0000 Subject: [PATCH 07/30] fix link --- website/docs/reference/resource-configs/tags.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index 6018c774282..992cc1ea22a 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -30,7 +30,7 @@ seeds: [](/reference/resource-configs/resource-path): +tags: | [] -[exports](/docs/use-dbt-semantic-layer/exports): +[exports](/docs/build/saved-queries#configure-exports): [](/reference/resource-configs/resource-path): +tags: | [] @@ -175,7 +175,7 @@ This following example shows how to apply tags to an export in the `dbt_project. ```yml -[exports](/docs/build/saved-queries-configure-exports): +[exports](/docs/build/saved-queries#configure-exports): jaffle_shop: customer_order_metrics: +tags: order_metrics From 8162530983af7e1be9457e7743d8c02993294ed1 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Mon, 18 Nov 2024 09:40:23 -0500 Subject: [PATCH 08/30] update tags --- website/docs/docs/build/saved-queries.md | 116 +++++++++++++++--- .../docs/reference/resource-configs/tags.md | 44 +++++-- 2 files changed, 138 insertions(+), 22 deletions(-) diff --git a/website/docs/docs/build/saved-queries.md b/website/docs/docs/build/saved-queries.md index 136474da3ee..68876b65160 100644 --- a/website/docs/docs/build/saved-queries.md +++ b/website/docs/docs/build/saved-queries.md @@ -17,8 +17,33 @@ To create a saved query, refer to the following table parameters. :::tip Note that we use the double colon (::) to indicate whether a parameter is nested within another parameter. So for example, `query_params::metrics` means the `metrics` parameter is nested under `query_params`. ::: + + + + +| Parameter | Type | Required | Description | +|-------|---------|----------|----------------| +| `name` | String | Required | Name of the saved query object. | +| `description` | String | Required | A description of the saved query. | +| `label` | String | Required | The display name for your saved query. This value will be shown in downstream tools. | +| `config` | String | Optional | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your saved query. Supports `cache`, [`enabled`](/reference/resource-configs/enabled), `export_as`, [`group`](/reference/resource-configs/group), [`meta`](/reference/resource-configs/meta), and [`schema`](/reference/resource-configs/schema) configurations. | +| `config::cache::enabled` | Object | Optional | An object with a sub-key used to specify if a saved query should populate the [cache](/docs/use-dbt-semantic-layer/sl-cache). Accepts sub-key `true` or `false`. Defaults to `false` | +| `query_params` | Structure | Required | Contains the query parameters. | +| `query_params::metrics` | List or String | Optional | A list of the metrics to be used in the query as specified in the command line interface. | +| `query_params::group_by` | List or String | Optional | A list of the Entities and Dimensions to be used in the query, which include the `Dimension` or `TimeDimension`. | +| `query_params::where` | List or String | Optional | A list of strings that may include the `Dimension` or `TimeDimension` objects. | +| `exports` | List or Structure | Optional | A list of exports to be specified within the exports structure. | +| `exports::name` | String | Required | Name of the export object. | +| `exports::config` | List or Structure | Required | A [`config`](/reference/resource-properties/config) property for any parameters specifying the export. | +| `exports::config::export_as` | String | Required | The type of export to run. Options include table or view currently and cache in the near future. | +| `exports::config::schema` | String | Optional | The [schema](/reference/resource-configs/schema) for creating the table or view. This option cannot be used for caching. | +| `exports::config::alias` | String | Optional | The table [alias](/reference/resource-configs/alias) used to write to the table or view. This option cannot be used for caching. | +| `exports::config::tags` | String | Optional | Apply a [tag](/reference/resource-configs/tags) (or list of tags) to a resource. Tags help organize and filter resources in dbt. Nested within `exports` and `config`. | + + + - + | Parameter | Type | Required | Description | |-------|---------|----------|----------------| @@ -33,15 +58,15 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `query_params::where` | List or String | Optional | A list of strings that may include the `Dimension` or `TimeDimension` objects. | | `exports` | List or Structure | Optional | A list of exports to be specified within the exports structure. | | `exports::name` | String | Required | Name of the export object. | -| `exports::config` | List or Structure | Required | A config section for any parameters specifying the export. | +| `exports::config` | List or Structure | Required | A [`config`](/reference/resource-properties/config) property for any parameters specifying the export. | | `exports::config::export_as` | String | Required | The type of export to run. Options include table or view currently and cache in the near future. | | `exports::config::schema` | String | Optional | The schema for creating the table or view. This option cannot be used for caching. | -| `exports::config::alias` | String | Optional | The table alias used to write to the table or view. This option cannot be used for caching. | +| `exports::config::alias` | String | Optional | The table alias used to write to the table or view. This option cannot be used for caching. | - + | Parameter | Type | Required | Description | |-------|---------|----------|----------------| @@ -54,11 +79,10 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `query_params::where` | List or String | Optional | Conditions nested with the `query_params`: a list of strings that may include the `Dimension` or `TimeDimension` objects. | | `exports` | List or Structure | Optional | A list of exports to be specified within the exports structure. | | `exports::name` | String | Required | Name of export object, nested within `exports`. | -| `exports::config` | List or Structure | Required | A config section for any parameters specifying the export, nested within `exports`. | +| `exports::config` | List or Structure | Required | A [`config`](/reference/resource-properties/config) property for any parameters specifying the export, nested within `exports`. | | `exports::config::export_as` | String | Required | Specifies the type of export: table, view, or upcoming cache options. Nested within `exports` and `config`. | | `exports::config::schema` | String | Optional | Schema for creating the table or view, not applicable for caching. Nested within `exports` and `config`. | | `exports::config::alias` | String | Optional | Table alias used to write to the table or view. This option can't be used for caching. Nested within `exports` and `config`. | -| `exports::config::tags` | String | Optional | Apply a [tag](/reference/resource-configs/tags) (or list of tags) to a resource. Tags help organize and filter resources in dbt. Nested within `exports` and `config`. | @@ -70,13 +94,12 @@ Use saved queries to define and manage common Semantic Layer queries in YAML, in In your saved query config, you can also leverage [caching](/docs/use-dbt-semantic-layer/sl-cache) with the dbt Cloud job scheduler to cache common queries, speed up performance, and reduce compute costs. - - - + In the following example, you can set the saved query in the `semantic_model.yml` file: + ```yaml saved_queries: @@ -97,13 +120,45 @@ saved_queries: exports: - name: my_export config: + export_as: table alias: my_export_alias + schema: my_export_schema_name + tags: 'my_tag' +``` + + + + + +```yaml +saved_queries: + - name: test_saved_query + description: "{{ doc('saved_query_description') }}" + label: Test saved query + config: + cache: + enabled: true # Or false if you want it disabled by default + query_params: + metrics: + - simple_metric + group_by: + - "Dimension('user__ds')" + where: + - "{{ Dimension('user__ds', 'DAY') }} <= now()" + - "{{ Dimension('user__ds', 'DAY') }} >= '2023-01-01'" + exports: + - name: my_export + config: export_as: table + alias: my_export_alias schema: my_export_schema_name - [tags](/reference/resource-configs/tags): 'my_tag' ``` + + + + Note, that you can set `export_as` to both the saved query and the exports [config](/reference/resource-properties/config), with the exports config value taking precedence. If a key isn't set in the exports config, it will inherit the saved query config value. #### Where clause @@ -123,7 +178,6 @@ filter: | filter: | {{ Metric('metric_name', group_by=['entity_name']) }} ``` - @@ -149,8 +203,8 @@ saved_queries: exports: - name: my_export config: - alias: my_export_alias export_as: table + alias: my_export_alias schema: my_export_schema_name ``` @@ -183,6 +237,38 @@ Once you've configured your saved query and set the foundation block, you can no The following is an example of a saved query with an export: + + +```yaml +saved_queries: + - name: order_metrics + description: Relevant order metrics + query_params: + metrics: + - orders + - large_order + - food_orders + - order_total + group_by: + - Entity('order_id') + - TimeDimension('metric_time', 'day') + - Dimension('customer__customer_name') + - ... # Additional group_by + where: + - "{{TimeDimension('metric_time')}} > current_timestamp - interval '1 week'" + - ... # Additional where clauses + exports: + - name: order_metrics + config: + export_as: table # Options available: table, view + [alias](/reference/resource-configs/alias): my_export_alias # Optional - defaults to Export name + [schema](/reference/resource-configs/schema): my_export_schema_name # Optional - defaults to deployment schema + [tags](/reference/resource-configs/tags): 'my_tag' + +``` + + + ```yaml saved_queries: @@ -206,9 +292,11 @@ saved_queries: - name: order_metrics config: export_as: table # Options available: table, view - schema: YOUR_SCHEMA # Optional - defaults to deployment schema - alias: SOME_TABLE_NAME # Optional - defaults to Export name + schema: my_export_schema_name # Optional - defaults to deployment schema + alias: my_export_alias # Optional - defaults to Export name ``` + + ## Run exports diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index 992cc1ea22a..e8838a72422 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -16,6 +16,8 @@ datatype: string | [string] + + ```yml models: @@ -30,11 +32,32 @@ seeds: [](/reference/resource-configs/resource-path): +tags: | [] -[exports](/docs/build/saved-queries#configure-exports): +``` + + + + +```yml + +models: + [](/reference/resource-configs/resource-path): + +tags: | [] + +snapshots: + [](/reference/resource-configs/resource-path): + +tags: | [] + +seeds: + [](/reference/resource-configs/resource-path): + +tags: | [] + +exports: [](/reference/resource-configs/resource-path): +tags: | [] ``` + + @@ -170,6 +193,14 @@ seeds: ### Apply tags to exports + + +Applying tags to exports is only available in dbt Core versions 1.9 and later. + + + + + This following example shows how to apply tags to an export in the `dbt_project.yml` file. The export is then tagged with `order_metrics`. @@ -203,11 +234,12 @@ saved_queries: ``` + ## Usage notes ### Tags are additive -Tags accumulate hierarchically. The above example would result in: +Tags accumulate hierarchically. The [earlier example](/reference/resource-configs/tags#use-tags-to-run-parts-of-your-project) would result in: | Model | Tags | | -------------------------------- | ------------------------------------- | @@ -218,7 +250,7 @@ Tags accumulate hierarchically. The above example would result in: ### Other resource types -Tags can also be applied to [sources](/docs/build/sources), [exposures](/docs/build/exposures), [exports](/docs/use-dbt-semantic-layer/exports), and even _specific columns_ in a resource. +Tags can also be applied to [sources](/docs/build/sources), [exposures](/docs/build/exposures), and even _specific columns_ in a resource. These resources do not yet support the `config` property, so you'll need to specify the tags as a top-level key instead. @@ -246,15 +278,11 @@ sources: tests: - unique: tags: ['test_level'] - -exports: - - name: export_name - tags: ['export_tag'] - ... ``` + In the example above, the `unique` test would be selected by any of these four tags: ```bash $ dbt test --select tag:top_level From 4cd118c64635203116552b90fe2c7c9f8ea1373b Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Mon, 18 Nov 2024 09:42:56 -0500 Subject: [PATCH 09/30] update rn --- website/docs/docs/dbt-versions/release-notes.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index f7faac30645..d96d5207bd3 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -19,12 +19,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo \* The official release date for this new format of release notes is May 15th, 2024. Historical release notes for prior dates may not reflect all available features released earlier this year or their tenancy availability. ## November 2024 -- **Fix**: This update improves [dbt Semantic Layer Tableau integration](/docs/cloud-integrations/semantic-layer/tableau) making query parsing more reliable. Some key fixes include: - - Error messages for unsupported joins between saved queries and ALL tables. - - Improved handling of queries when multiple tables are selected in a data source. - - Fixed a bug when an IN filter contained a lot of values. - - Better error messaging for queries that can't be parsed correctly. -- **New**: Exports now support [tags](/reference/resource-configs/tags) in dbt Cloud. Tags allow you to categorize your resources and filter them in dbt Cloud. You can add tags to your [exports](/docs/build/saved-queries#configure-exports) in the `semantic_model.yml` file or `dbt_project.yml` file. For example: +- **New**: Exports now support [tags](/reference/resource-configs/tags) in dbt. Tags allow you to categorize your resources and filter them. You can add tags to your [exports](/docs/build/saved-queries#configure-exports) in the `semantic_model.yml` file or `dbt_project.yml` file. For example: ```yml exports: @@ -32,6 +27,12 @@ Release notes are grouped by month for both multi-tenant and virtual private clo tags: ['export_tag'] ... ``` +- **Fix**: This update improves [dbt Semantic Layer Tableau integration](/docs/cloud-integrations/semantic-layer/tableau) making query parsing more reliable. Some key fixes include: + - Error messages for unsupported joins between saved queries and ALL tables. + - Improved handling of queries when multiple tables are selected in a data source. + - Fixed a bug when an IN filter contained a lot of values. + - Better error messaging for queries that can't be parsed correctly. + ## October 2024 From c34e68e443bf544bd6b0fd1565eb9612d7ddd2fe Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Mon, 18 Nov 2024 15:24:53 -0500 Subject: [PATCH 10/30] update meta --- website/docs/docs/build/conversion-metrics.md | 12 +- website/docs/docs/build/dimensions.md | 107 +++++++++++++- website/docs/docs/build/entities.md | 69 ++++++++- website/docs/docs/build/measures.md | 136 ++++++++++++++++- website/docs/docs/build/simple.md | 4 +- .../docs/reference/resource-configs/meta.md | 138 ++++++++++++++---- website/snippets/_sl-measures-parameters.md | 1 + 7 files changed, 416 insertions(+), 51 deletions(-) diff --git a/website/docs/docs/build/conversion-metrics.md b/website/docs/docs/build/conversion-metrics.md index 2ef2c3910b9..979ee03e7b9 100644 --- a/website/docs/docs/build/conversion-metrics.md +++ b/website/docs/docs/build/conversion-metrics.md @@ -31,13 +31,13 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `entity` | The entity for each conversion event. | Required | | `calculation` | Method of calculation. Either `conversion_rate` or `conversions`. Defaults to `conversion_rate`. | Optional | | `base_measure` | A list of base measure inputs | Required | -| `base_measure:name` | The base conversion event measure. | Required | -| `base_measure:fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional | -| `base_measure:join_to_timespine` | Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. | Optional | +| `base_measure::name` | The base conversion event measure. | Required | +| `base_measure::fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional | +| `base_measure::join_to_timespine` | Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. | Optional | | `conversion_measure` | A list of conversion measure inputs. | Required | -| `conversion_measure:name` | The base conversion event measure.| Required | -| `conversion_measure:fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional | -| `conversion_measure:join_to_timespine` | Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. | Optional | +| `conversion_measure::name` | The base conversion event measure.| Required | +| `conversion_measure::fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional | +| `conversion_measure::join_to_timespine` | Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. | Optional | | `window` | The time window for the conversion event, such as 7 days, 1 week, 3 months. Defaults to infinity. | Optional | | `constant_properties` | List of constant properties. | Optional | | `base_property` | The property from the base semantic model that you want to hold constant. | Optional | diff --git a/website/docs/docs/build/dimensions.md b/website/docs/docs/build/dimensions.md index eb1809ea72b..424fd72e222 100644 --- a/website/docs/docs/build/dimensions.md +++ b/website/docs/docs/build/dimensions.md @@ -14,6 +14,26 @@ Groups are defined within semantic models, alongside entities and measures, and All dimensions require a `name`, `type`, and can optionally include an `expr` parameter. The `name` for your Dimension must be unique within the same semantic model. +:::tip +Note that we use the double colon (::) to indicate whether a parameter is nested within another parameter. So for example,` config::config_name` means the `config_name` parameter is nested under `config`. +::: + + + +| Parameter | Description | Type | +| --------- | ----------- | ---- | +| `name` | Refers to the name of the group that will be visible to the user in downstream tools. It can also serve as an alias if the column name or SQL query reference is different and provided in the `expr` parameter.

Dimension names should be unique within a semantic model, but they can be non-unique across different models as MetricFlow uses [joins](/docs/build/join-logic) to identify the right dimension. | Required | +| `type` | Specifies the type of group created in the semantic model. There are two types:

- **Categorical**: Describe attributes or features like geography or sales region.
- **Time**: Time-based dimensions like timestamps or dates. | Required | +| `type_params` | Specific type params such as if the time is primary or used as a partition | Required | +| `description` | A clear description of the dimension | Optional | +| `expr` | Defines the underlying column or SQL query for a dimension. If no `expr` is specified, MetricFlow will use the column with the same name as the group. You can use the column name itself to input a SQL expression. | Optional | +| `label` | A recommended string that defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | +| `config` | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your metric. Supports [`meta`](/reference/resource-configs/meta), [`group`](/reference/resource-configs/group), and [`enabled`](/reference/resource-configs/enabled) configs. | Optional | + + + + + | Parameter | Description | Type | | --------- | ----------- | ---- | | `name` | Refers to the name of the group that will be visible to the user in downstream tools. It can also serve as an alias if the column name or SQL query reference is different and provided in the `expr` parameter.

Dimension names should be unique within a semantic model, but they can be non-unique across different models as MetricFlow uses [joins](/docs/build/join-logic) to identify the right dimension. | Required | @@ -22,7 +42,10 @@ All dimensions require a `name`, `type`, and can optionally include an `expr` pa | `description` | A clear description of the dimension | Optional | | `expr` | Defines the underlying column or SQL query for a dimension. If no `expr` is specified, MetricFlow will use the column with the same name as the group. You can use the column name itself to input a SQL expression. | Optional | | `label` | A recommended string that defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | -| `meta` | Use the [`meta` field](/reference/resource-configs/meta) to set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | +| `config` | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your metric. Supports [`meta`](/reference/resource-configs/meta), [`group`](/reference/resource-configs/group), and [`enabled`](/reference/resource-configs/enabled) configs. | Optional | +| `config::meta` | Use [`meta`](/reference/resource-configs/meta), nested under `config`, to set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | + + Refer to the following for the complete specification for dimensions: @@ -38,6 +61,8 @@ dimensions: Refer to the following example to see how dimensions are used in a semantic model: + + ```yaml semantic_models: - name: transactions @@ -69,6 +94,40 @@ semantic_models: - name: type type: categorical ``` + + + + +```yaml +semantic_models: + - name: transactions + description: A record for every transaction that takes place. Carts are considered multiple transactions for each SKU. + model: {{ ref('fact_transactions') }} + defaults: + agg_time_dimension: order_date +# --- entities --- + entities: + - name: transaction + type: primary + ... +# --- measures --- + measures: + ... +# --- dimensions --- + dimensions: + - name: order_date + type: time + type_params: + time_granularity: day + label: "Date of transaction" # Recommend adding a label to provide more context to users consuming the data + expr: ts + - name: is_bulk + type: categorical + expr: case when quantity > 10 then true else false end + - name: type + type: categorical +``` + Dimensions are bound to the primary entity of the semantic model they are defined in. For example the dimension `type` is defined in a model that has `transaction` as a primary entity. `type` is scoped to the `transaction` entity, and to reference this dimension you would use the fully qualified dimension name i.e `transaction__type`. @@ -104,6 +163,8 @@ This section further explains the dimension definitions, along with examples. Di Categorical dimensions are used to group metrics by different attributes, features, or characteristics such as product type. They can refer to existing columns in your dbt model or be calculated using a SQL expression with the `expr` parameter. An example of a categorical dimension is `is_bulk_transaction`, which is a group created by applying a case statement to the underlying column `quantity`. This allows users to group or filter the data based on bulk transactions. + + ```yaml dimensions: - name: is_bulk_transaction @@ -112,6 +173,17 @@ dimensions: meta: usage: "Filter to identify bulk transactions, like where quantity > 10." ``` + + + + +```yaml +dimensions: + - name: is_bulk_transaction + type: categorical + expr: case when quantity > 10 then true else false end +``` + ## Time @@ -135,6 +207,8 @@ You can set `is_partition` for time to define specific time spans. Additionally, Use `is_partition: True` to show that a dimension exists over a specific time window. For example, a date-partitioned dimensional table. When you query metrics from different tables, the dbt Semantic Layer uses this parameter to ensure that the correct dimensional values are joined to measures. + + ```yaml dimensions: - name: created_at @@ -163,6 +237,37 @@ measures: expr: 1 agg: sum ``` + + + + +```yaml +dimensions: + - name: created_at + type: time + label: "Date of creation" + expr: ts_created # ts_created is the underlying column name from the table + is_partition: True + type_params: + time_granularity: day + - name: deleted_at + type: time + label: "Date of deletion" + expr: ts_deleted # ts_deleted is the underlying column name from the table + is_partition: True + type_params: + time_granularity: day + +measures: + - name: users_deleted + expr: 1 + agg: sum + agg_time_dimension: deleted_at + - name: users_created + expr: 1 + agg: sum +``` + diff --git a/website/docs/docs/build/entities.md b/website/docs/docs/build/entities.md index e4ed0773c3c..558dfd3aea4 100644 --- a/website/docs/docs/build/entities.md +++ b/website/docs/docs/build/entities.md @@ -95,17 +95,67 @@ Natural keys are columns or combinations of columns in a table that uniquely ide The following is the complete spec for entities: + + +```yaml +semantic_models: + - name: semantic_model_name + ..rest of the semantic model config + entities: + - name: entity_name ## Required + type: Primary, natural, foreign, or unique ## Required + description: A description of the field or role the entity takes in this table ## Optional + expr: The field that denotes that entity (transaction_id). ## Optional + Defaults to name if unspecified. + [config](/reference/resource-properties/config): Specify configurations for entity. ## Optional + [meta](/reference/resource-configs/meta): {} Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. ## Optional +``` + + + + +```yaml +semantic_models: + - name: semantic_model_name + ..rest of the semantic model config + entities: + - name: entity_name ## Required + type: Primary, or natural, or foreign, or unique ## Required + description: A description of the field or role the entity takes in this table ## Optional + expr: The field that denotes that entity (transaction_id). ## Optional + Defaults to name if unspecified. +``` + + +Here's an example of how to define entities in a semantic model: + + + ```yaml entities: - - name: transaction ## Required - type: Primary or natural or foreign or unique ## Required + - name: transaction + type: primary + expr: id_transaction + - name: order + type: foreign + expr: id_order + - name: user + type: foreign + expr: substring(id_order from 2) + entities: + - name: transaction + type: description: A description of the field or role the entity takes in this table ## Optional - expr: The field that denotes that entity (transaction_id). ## Optional + expr: The field that denotes that entity (transaction_id). Defaults to name if unspecified. + [config](/reference/resource-properties/config): + [meta](/reference/resource-configs/meta): + data_owner: "Finance team" ``` + + + -Here's an example of how to define entities in a semantic model: - ```yaml entities: - name: transaction @@ -117,11 +167,18 @@ entities: - name: user type: foreign expr: substring(id_order from 2) + entities: + - name: transaction + type: + description: A description of the field or role the entity takes in this table ## Optional + expr: The field that denotes that entity (transaction_id). + Defaults to name if unspecified. ``` + ## Combine columns with a key -If a table doesn't have any key (like a primary key), use _surrogate combination_ to form a key that will help you identify a record by combining two columns. This applies to any [entity type](/docs/build/entities#entity-types). For example, you can combine `date_key` and `brand_code` from the `raw_brand_target_weekly` table to form a _surrogate key_. The following example creates a surrogate key by joining `date_key` and `brand_code` using a pipe (`|`) as a separator. +If a table doesn't have any key (like a primary key), use _surrogate combination_ to form a key that will help you identify a record by combining two columns. This applies to any [entity type](/docs/build/entities#entity-types). For example, you can combine `date_key` and `brand_code` from the `raw_brand_target_weekly` table to form a _surrogate key_. The following example creates a surrogate key by joining `date_key` and `brand_code` using a pipe (`|`) as a separator. ```yaml diff --git a/website/docs/docs/build/measures.md b/website/docs/docs/build/measures.md index 977b630fada..a1470e72be9 100644 --- a/website/docs/docs/build/measures.md +++ b/website/docs/docs/build/measures.md @@ -10,6 +10,10 @@ Measures are aggregations performed on columns in your model. They can be used a Measures have several inputs, which are described in the following table along with their field types. +:::tip +Note that we use the double colon (::) to indicate whether a parameter is nested within another parameter. So for example,` config::config_name` means the `config_name` parameter is nested under `config`. +::: + import MeasuresParameters from '/snippets/_sl-measures-parameters.md'; @@ -18,16 +22,41 @@ import MeasuresParameters from '/snippets/_sl-measures-parameters.md'; An example of the complete YAML measures spec is below. The actual configuration of your measures will depend on the aggregation you're using. + + +```yaml +semantic_models: + - name: semantic_model_name + ..rest of the semantic model config + measures: + - name: The name of the measure + description: 'same as always' ## Optional + agg: the aggregation type. + expr: the field + agg_params: 'specific aggregation properties such as a percentile' ## Optional + agg_time_dimension: The time field. Defaults to the default agg time dimension for the semantic model. ## Optional + non_additive_dimension: 'Use these configs when you need non-additive dimensions.' ## Optional + [config](/reference/resource-properties/config): Use the config property to specify configurations for your measure. ## Optional + [meta](/reference/resource-configs/meta): {} Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. ## Optional +``` + + + + ```yaml -measures: - - name: The name of the measure - description: 'same as always' ## Optional - agg: the aggregation type. - expr: the field - agg_params: 'specific aggregation properties such as a percentile' ## Optional - agg_time_dimension: The time field. Defaults to the default agg time dimension for the semantic model. ## Optional - non_additive_dimension: 'Use these configs when you need non-additive dimensions.' ## Optional +semantic_models: + - name: semantic_model_name + ..rest of the semantic model config + measures: + - name: The name of the measure + description: 'same as always' ## Optional + agg: the aggregation type. + expr: the field + agg_params: 'specific aggregation properties such as a percentile' ## Optional + agg_time_dimension: The time field. Defaults to the default agg time dimension for the semantic model. ## Optional + non_additive_dimension: 'Use these configs when you need non-additive dimensions.' ## Optional ``` + ### Name @@ -96,6 +125,96 @@ If you use the `dayofweek` function in the `expr` parameter with the legacy Snow ### Model with different aggregations + + +```yaml +semantic_models: + - name: transactions + description: A record of every transaction that takes place. Carts are considered multiple transactions for each SKU. + model: ref('schema.transactions') + defaults: + agg_time_dimension: transaction_date + +# --- entities --- + entities: + - name: transaction_id + type: primary + - name: customer_id + type: foreign + - name: store_id + type: foreign + - name: product_id + type: foreign + +# --- measures --- + measures: + - name: transaction_amount_usd + description: Total USD value of transactions + expr: transaction_amount_usd + agg: sum + config: + meta: + used_in_reporting: true + - name: transaction_amount_usd_avg + description: Average USD value of transactions + expr: transaction_amount_usd + agg: average + - name: transaction_amount_usd_max + description: Maximum USD value of transactions + expr: transaction_amount_usd + agg: max + - name: transaction_amount_usd_min + description: Minimum USD value of transactions + expr: transaction_amount_usd + agg: min + - name: quick_buy_transactions + description: The total transactions bought as quick buy + expr: quick_buy_flag + agg: sum_boolean + - name: distinct_transactions_count + description: Distinct count of transactions + expr: transaction_id + agg: count_distinct + - name: transaction_amount_avg + description: The average value of transactions + expr: transaction_amount_usd + agg: average + - name: transactions_amount_usd_valid # Notice here how we use expr to compute the aggregation based on a condition + description: The total USD value of valid transactions only + expr: CASE WHEN is_valid = True then transaction_amount_usd else 0 end + agg: sum + - name: transactions + description: The average value of transactions. + expr: transaction_amount_usd + agg: average + - name: p99_transaction_value + description: The 99th percentile transaction value + expr: transaction_amount_usd + agg: percentile + agg_params: + percentile: .99 + use_discrete_percentile: False # False calculates the continuous percentile, True calculates the discrete percentile. + - name: median_transaction_value + description: The median transaction value + expr: transaction_amount_usd + agg: median + +# --- dimensions --- + dimensions: + - name: transaction_date + type: time + expr: date_trunc('day', ts) # expr refers to underlying column ts + type_params: + time_granularity: day + - name: is_bulk_transaction + type: categorical + expr: case when quantity > 10 then true else false end + +``` + + + + ```yaml semantic_models: - name: transactions @@ -177,6 +296,7 @@ semantic_models: expr: case when quantity > 10 then true else false end ``` + ### Non-additive dimensions diff --git a/website/docs/docs/build/simple.md b/website/docs/docs/build/simple.md index f57d498d290..4a3aac32708 100644 --- a/website/docs/docs/build/simple.md +++ b/website/docs/docs/build/simple.md @@ -24,8 +24,8 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `type_params` | The type parameters of the metric. | Required | | `measure` | A list of measure inputs | Required | | `measure:name` | The measure you're referencing. | Required | -| `measure:fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional | -| `measure:join_to_timespine` | Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. | Optional | +| `measure::fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional | +| `measure::join_to_timespine` | Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. | Optional | The following displays the complete specification for simple metrics, along with an example. diff --git a/website/docs/reference/resource-configs/meta.md b/website/docs/reference/resource-configs/meta.md index fa0d48c299e..3d861fc5360 100644 --- a/website/docs/reference/resource-configs/meta.md +++ b/website/docs/reference/resource-configs/meta.md @@ -16,10 +16,9 @@ hide_table_of_contents: true { label: 'Analyses', value: 'analyses', }, { label: 'Macros', value: 'macros', }, { label: 'Exposures', value: 'exposures', }, - { label: 'Semantic Models', value: 'semantic models', }, + { label: 'Semantic models', value: 'semantic models', }, { label: 'Metrics', value: 'metrics', }, { label: 'Saved queries', value: 'saved queries', }, - { label: 'Dimensions', value: 'dimensions', }, ] }> @@ -180,6 +179,10 @@ exposures: +Configure `meta` in the your semantic models YAML file or under the `semantic-models` config block in the `dbt_project.yml` file. + + + ```yml @@ -191,6 +194,39 @@ semantic_models: ``` + + + + +[Dimensions](/docs/build/dimensions), [entities](/docs/build/entities), and [measures](/docs/build/measures) can also have their own `meta` configurations. + + + +```yml +semantic_models: + - name: semantic_model_name + config: + meta: {} + + dimensions: + - name: dimension_name + config: + meta: {} + + entities: + - name: entity_name + config: + meta: {} + + measures: + - name: measure_name + config: + meta: {} + +``` + + + The `meta` config can also be defined under the `semantic-models` config block in `dbt_project.yml`. See [configs and properties](/reference/configs-and-properties) for details. @@ -250,25 +286,7 @@ saved_queries: ``` - - - - - -```yml -semantic_models: - - name: semantic_model - ... - [dimensions](/docs/build/dimensions): - - name: order_date - config: - meta: {} -``` - - - - ## Definition @@ -360,9 +378,54 @@ models: -### Assign owner and additional metadata to dimensions +### Assign meta to semantic model + -The following example shows how to assign a `data_owner` and additional metadata value to a dimension in a semantic model. +The following example shows how to assign a `meta` value to a semantic model in the `semantic_model.yml` file and `dbt_project.yml` file: + + + + +```yaml +semantic_models: + - name: transaction + model: ref('fact_transactions') + description: "Transaction fact table at the transaction level. This table contains one row per transaction and includes the transaction timestamp." + defaults: + agg_time_dimension: transaction_date + config: + meta: + data_owner: "Finance team" + used_in_reporting: true +``` + + + + + +```yaml +semantic-models: + jaffle_shop: + +meta: + used_in_reporting: true +``` + + + +### Assign meta to dimensions, measures, entities + + + +Available in dbt version 1.9 and later. + + + + + + + + +The following example shows how to assign a `meta` value to a [dimension](/docs/build/dimensions), [entity](/docs/build/entities), and [measure](/docs/build/measures) in a semantic model: @@ -370,16 +433,32 @@ The following example shows how to assign a `data_owner` and additional metadata semantic_models: - name: semantic_model ... - [dimensions](/docs/build/dimensions): - - name: order_date - meta: - data_owner: "Finance team" - used_in_reporting: true + dimensions: + - name: order_date + type: time + meta: + data_owner: "Finance team" + used_in_reporting: true + entities: + - name: customer_id + type: primary + meta: + description: "Unique identifier for customers" + data_owner: "Sales team" + used_in_reporting: false + measures: + - name: count_of_users + expr: user_id + meta: + used_in_reporting: true ``` + + + -This second example shows how to assign a `data_owner` and additional metadata value to a dimension in the `dbt_project.yml` file using the `+meta` syntax. +This second example shows how to assign a `data_owner` and additional metadata value to a dimension in the `dbt_project.yml` file using the `+meta` syntax. The similar syntax can be used for entities and measures. @@ -395,3 +474,6 @@ semantic-models: ``` + + + diff --git a/website/snippets/_sl-measures-parameters.md b/website/snippets/_sl-measures-parameters.md index 728d63c6b4f..09916714b96 100644 --- a/website/snippets/_sl-measures-parameters.md +++ b/website/snippets/_sl-measures-parameters.md @@ -9,3 +9,4 @@ | `agg_time_dimension` | The time field. Defaults to the default agg time dimension for the semantic model. | Optional | 1.6 and higher | | `label` | String that defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as orders_total or "orders_total"). Available in dbt version 1.7 or higher. | Optional | `create_metric` | Create a `simple` metric from a measure by setting `create_metric: True`. The `label` and `description` attributes will be automatically propagated to the created metric. Available in dbt version 1.7 or higher. | Optional | +| `config::meta` | Use [`meta`](/reference/resource-configs/meta), nested under [`config`](/reference/resource-properties/config), to set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | From 7824b08ee66b24685b0caa9550fdf17ebf6bff83 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Mon, 18 Nov 2024 16:32:51 -0500 Subject: [PATCH 11/30] update --- website/docs/docs/build/dimensions.md | 9 ++------- website/docs/docs/build/measures.md | 4 ---- website/snippets/_sl-measures-parameters.md | 2 +- 3 files changed, 3 insertions(+), 12 deletions(-) diff --git a/website/docs/docs/build/dimensions.md b/website/docs/docs/build/dimensions.md index 424fd72e222..a53869bef3c 100644 --- a/website/docs/docs/build/dimensions.md +++ b/website/docs/docs/build/dimensions.md @@ -14,10 +14,6 @@ Groups are defined within semantic models, alongside entities and measures, and All dimensions require a `name`, `type`, and can optionally include an `expr` parameter. The `name` for your Dimension must be unique within the same semantic model. -:::tip -Note that we use the double colon (::) to indicate whether a parameter is nested within another parameter. So for example,` config::config_name` means the `config_name` parameter is nested under `config`. -::: - | Parameter | Description | Type | @@ -28,7 +24,7 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `description` | A clear description of the dimension | Optional | | `expr` | Defines the underlying column or SQL query for a dimension. If no `expr` is specified, MetricFlow will use the column with the same name as the group. You can use the column name itself to input a SQL expression. | Optional | | `label` | A recommended string that defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | -| `config` | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your metric. Supports [`meta`](/reference/resource-configs/meta), [`group`](/reference/resource-configs/group), and [`enabled`](/reference/resource-configs/enabled) configs. | Optional | +| `config` | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your metric. Supports the [`meta`](/reference/resource-configs/meta) property, nested under `config`. | Optional | @@ -42,8 +38,7 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `description` | A clear description of the dimension | Optional | | `expr` | Defines the underlying column or SQL query for a dimension. If no `expr` is specified, MetricFlow will use the column with the same name as the group. You can use the column name itself to input a SQL expression. | Optional | | `label` | A recommended string that defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | -| `config` | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your metric. Supports [`meta`](/reference/resource-configs/meta), [`group`](/reference/resource-configs/group), and [`enabled`](/reference/resource-configs/enabled) configs. | Optional | -| `config::meta` | Use [`meta`](/reference/resource-configs/meta), nested under `config`, to set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | +| `config` | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your metric. Supports the [`meta`](/reference/resource-configs/meta) configs. | Optional | diff --git a/website/docs/docs/build/measures.md b/website/docs/docs/build/measures.md index a1470e72be9..c12cf67fdc3 100644 --- a/website/docs/docs/build/measures.md +++ b/website/docs/docs/build/measures.md @@ -10,10 +10,6 @@ Measures are aggregations performed on columns in your model. They can be used a Measures have several inputs, which are described in the following table along with their field types. -:::tip -Note that we use the double colon (::) to indicate whether a parameter is nested within another parameter. So for example,` config::config_name` means the `config_name` parameter is nested under `config`. -::: - import MeasuresParameters from '/snippets/_sl-measures-parameters.md'; diff --git a/website/snippets/_sl-measures-parameters.md b/website/snippets/_sl-measures-parameters.md index 09916714b96..8058474d1ef 100644 --- a/website/snippets/_sl-measures-parameters.md +++ b/website/snippets/_sl-measures-parameters.md @@ -9,4 +9,4 @@ | `agg_time_dimension` | The time field. Defaults to the default agg time dimension for the semantic model. | Optional | 1.6 and higher | | `label` | String that defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as orders_total or "orders_total"). Available in dbt version 1.7 or higher. | Optional | `create_metric` | Create a `simple` metric from a measure by setting `create_metric: True`. The `label` and `description` attributes will be automatically propagated to the created metric. Available in dbt version 1.7 or higher. | Optional | -| `config::meta` | Use [`meta`](/reference/resource-configs/meta), nested under [`config`](/reference/resource-properties/config), to set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | +| `config` | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your metric. Supports the [`meta`](/reference/resource-configs/meta) property, nested under `config`. | Optional | From 16faf8cde0cd3659f8d300a81214f7c7be61f7dd Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 10 Dec 2024 10:47:54 +0000 Subject: [PATCH 12/30] Update exports.md --- website/docs/docs/use-dbt-semantic-layer/exports.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/website/docs/docs/use-dbt-semantic-layer/exports.md b/website/docs/docs/use-dbt-semantic-layer/exports.md index d5285c8f779..2ecffc508a2 100644 --- a/website/docs/docs/use-dbt-semantic-layer/exports.md +++ b/website/docs/docs/use-dbt-semantic-layer/exports.md @@ -24,10 +24,7 @@ Essentially, exports are like any other table in your data platform — they ## Benefits of exports -The following section explains the main benefits of using exports, including: -- [DRY representation](#dry-representation) -- [Easier changes](#easier-changes) -- [Caching](#caching) +The following section explains the main benefits of using exports: From 7118aa5b95586e7c6beacc67617868aa867e2314 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 10 Dec 2024 10:54:15 +0000 Subject: [PATCH 13/30] Update dimensions.md --- website/docs/docs/build/dimensions.md | 1 + 1 file changed, 1 insertion(+) diff --git a/website/docs/docs/build/dimensions.md b/website/docs/docs/build/dimensions.md index 70715537f9a..f1a2f10bf89 100644 --- a/website/docs/docs/build/dimensions.md +++ b/website/docs/docs/build/dimensions.md @@ -22,6 +22,7 @@ All dimensions require a `name`, `type`, and can optionally include an `expr` pa | `description` | A clear description of the dimension. | Optional | String | | `expr` | Defines the underlying column or SQL query for a dimension. If no `expr` is specified, MetricFlow will use the column with the same name as the group. You can use the column name itself to input a SQL expression. | Optional | String | | `label` | Defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String | +| [`meta`](/reference/resource-configs/meta) | Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. | Optional | {} | Refer to the following for the complete specification for dimensions: From e7f407f013642f639012c34a7560a6941c6d0906 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 10 Dec 2024 10:59:47 +0000 Subject: [PATCH 14/30] Update website/docs/docs/build/dimensions.md --- website/docs/docs/build/dimensions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/dimensions.md b/website/docs/docs/build/dimensions.md index f1a2f10bf89..609b2168906 100644 --- a/website/docs/docs/build/dimensions.md +++ b/website/docs/docs/build/dimensions.md @@ -22,7 +22,7 @@ All dimensions require a `name`, `type`, and can optionally include an `expr` pa | `description` | A clear description of the dimension. | Optional | String | | `expr` | Defines the underlying column or SQL query for a dimension. If no `expr` is specified, MetricFlow will use the column with the same name as the group. You can use the column name itself to input a SQL expression. | Optional | String | | `label` | Defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String | -| [`meta`](/reference/resource-configs/meta) | Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. | Optional | {} | +| [`meta`](/reference/resource-configs/meta) | Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. | Optional | | Refer to the following for the complete specification for dimensions: From 5617cf82b9666c01087d5a0c7d3afa1f63d60ded Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 10 Dec 2024 11:18:14 +0000 Subject: [PATCH 15/30] Update dimensions.md --- website/docs/docs/build/dimensions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/dimensions.md b/website/docs/docs/build/dimensions.md index 609b2168906..4ad06e5626e 100644 --- a/website/docs/docs/build/dimensions.md +++ b/website/docs/docs/build/dimensions.md @@ -22,7 +22,7 @@ All dimensions require a `name`, `type`, and can optionally include an `expr` pa | `description` | A clear description of the dimension. | Optional | String | | `expr` | Defines the underlying column or SQL query for a dimension. If no `expr` is specified, MetricFlow will use the column with the same name as the group. You can use the column name itself to input a SQL expression. | Optional | String | | `label` | Defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String | -| [`meta`](/reference/resource-configs/meta) | Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. | Optional | | +| [`meta`](/reference/resource-configs/meta) | Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. | Optional | Dictionary | Refer to the following for the complete specification for dimensions: From e2ccfb34d341c3bf3e8262ad02062a2e2164626d Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 11 Dec 2024 11:27:31 +0000 Subject: [PATCH 16/30] update to saved queries --- website/docs/docs/build/saved-queries.md | 14 ++-- .../docs/docs/dbt-versions/release-notes.md | 12 +-- .../docs/reference/resource-configs/tags.md | 77 ++++++++++--------- 3 files changed, 54 insertions(+), 49 deletions(-) diff --git a/website/docs/docs/build/saved-queries.md b/website/docs/docs/build/saved-queries.md index 68876b65160..840b1ebb95c 100644 --- a/website/docs/docs/build/saved-queries.md +++ b/website/docs/docs/build/saved-queries.md @@ -26,7 +26,7 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `name` | String | Required | Name of the saved query object. | | `description` | String | Required | A description of the saved query. | | `label` | String | Required | The display name for your saved query. This value will be shown in downstream tools. | -| `config` | String | Optional | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your saved query. Supports `cache`, [`enabled`](/reference/resource-configs/enabled), `export_as`, [`group`](/reference/resource-configs/group), [`meta`](/reference/resource-configs/meta), and [`schema`](/reference/resource-configs/schema) configurations. | +| `config` | String | Optional | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your saved query. Supports `cache`, [`enabled`](/reference/resource-configs/enabled), `export_as`, [`group`](/reference/resource-configs/group), [`meta`](/reference/resource-configs/meta), [`tags`](/reference/resource-configs/tags), and [`schema`](/reference/resource-configs/schema) configurations. | | `config::cache::enabled` | Object | Optional | An object with a sub-key used to specify if a saved query should populate the [cache](/docs/use-dbt-semantic-layer/sl-cache). Accepts sub-key `true` or `false`. Defaults to `false` | | `query_params` | Structure | Required | Contains the query parameters. | | `query_params::metrics` | List or String | Optional | A list of the metrics to be used in the query as specified in the command line interface. | @@ -38,7 +38,6 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `exports::config::export_as` | String | Required | The type of export to run. Options include table or view currently and cache in the near future. | | `exports::config::schema` | String | Optional | The [schema](/reference/resource-configs/schema) for creating the table or view. This option cannot be used for caching. | | `exports::config::alias` | String | Optional | The table [alias](/reference/resource-configs/alias) used to write to the table or view. This option cannot be used for caching. | -| `exports::config::tags` | String | Optional | Apply a [tag](/reference/resource-configs/tags) (or list of tags) to a resource. Tags help organize and filter resources in dbt. Nested within `exports` and `config`. | @@ -108,7 +107,8 @@ saved_queries: label: Test saved query config: cache: - enabled: true # Or false if you want it disabled by default + [enabled](/reference/resource-configs/enabled): true | false + [tags](/reference/resource-configs/tags): 'my_tag' query_params: metrics: - simple_metric @@ -123,7 +123,6 @@ saved_queries: export_as: table alias: my_export_alias schema: my_export_schema_name - tags: 'my_tag' ``` @@ -243,6 +242,9 @@ The following is an example of a saved query with an export: saved_queries: - name: order_metrics description: Relevant order metrics + config: + tags: + - order_metrics query_params: metrics: - orders @@ -262,9 +264,7 @@ saved_queries: config: export_as: table # Options available: table, view [alias](/reference/resource-configs/alias): my_export_alias # Optional - defaults to Export name - [schema](/reference/resource-configs/schema): my_export_schema_name # Optional - defaults to deployment schema - [tags](/reference/resource-configs/tags): 'my_tag' - + [schema](/reference/resource-configs/schema): my_export_schema_name # Optional - defaults to deployment schema ``` diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index ad8e2d3a9a9..88d3d2b88f2 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -20,14 +20,16 @@ Release notes are grouped by month for both multi-tenant and virtual private clo ## December 2024 -- **New**: Exports now support [tags](/reference/resource-configs/tags) in dbt. Tags allow you to categorize your resources and filter them. You can add tags to your [exports](/docs/build/saved-queries#configure-exports) in the `semantic_model.yml` file or `dbt_project.yml` file. For example: +- **New**: Saved queries now support [tags](/reference/resource-configs/tags), which allow you to categorize your resources and filter them. Add tags to your [saved queries](/docs/build/saved-queries) in the `semantic_model.yml` file or `dbt_project.yml` file. For example: + ```yml - exports: - - name: export_name - tags: ['export_tag'] - ... + [saved-queries](/docs/build/saved-queries): + jaffle_shop: + customer_order_metrics: + +tags: order_metrics ``` + - **New**: [Model notifications](/docs/deploy/model-notifications) are now generally available in dbt Cloud. These notifications alert model owners through email about any issues encountered by models and tests as soon as they occur while running a job. - **New**: You can now use your [Azure OpenAI key](/docs/cloud/account-integrations?ai-integration=azure#ai-integrations) (available in beta) to use dbt Cloud features like [dbt Copilot](/docs/cloud/dbt-copilot) and [Ask dbt](/docs/cloud-integrations/snowflake-native-app) . Additionally, you can use your own [OpenAI API key](/docs/cloud/account-integrations?ai-integration=openai#ai-integrations) or use [dbt Labs-managed OpenAI](/docs/cloud/account-integrations?ai-integration=dbtlabs#ai-integrations) key. Refer to [AI integrations](/docs/cloud/account-integrations#ai-integrations) for more information. - **New**: The [`hard_deletes`](/reference/resource-configs/hard-deletes) config gives you more control on how to handle deleted rows from the source. Supported options are `ignore` (default), `invalidate` (replaces the legacy `invalidate_hard_deletes=true`), and `new_record`. Note that `new_record` will create a new metadata column in the snapshot table. diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index e8838a72422..45c4e719059 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -20,15 +20,15 @@ datatype: string | [string] ```yml -models: +[models](/reference/model-configs): [](/reference/resource-configs/resource-path): +tags: | [] -snapshots: +[snapshots](/reference/snapshot-configs): [](/reference/resource-configs/resource-path): +tags: | [] -seeds: +[seeds](/reference/seed-configs): [](/reference/resource-configs/resource-path): +tags: | [] @@ -39,19 +39,19 @@ seeds: ```yml -models: +[models](/reference/model-configs): [](/reference/resource-configs/resource-path): +tags: | [] -snapshots: +[snapshots](/reference/snapshot-configs): [](/reference/resource-configs/resource-path): +tags: | [] -seeds: +[seeds](/reference/seed-configs): [](/reference/resource-configs/resource-path): +tags: | [] -exports: +[saved-queries:](/docs/build/saved-queries) [](/reference/resource-configs/resource-path): +tags: | [] @@ -64,38 +64,45 @@ exports: - + -```yml -version: 2 +The following examples show how to add tags to dbt resources in YAML files. Replace `resource_type` with `models`, `snapshots`, `seeds`, or `saved_queries` as appropriate. + -models: - - name: model_name - config: - tags: | [] + + +The following examples show how to add tags to dbt resources in YAML files. Replace `resource_type` with `models`, `snapshots`, or `seeds` as appropriate. + + + +```yaml +resource_type: + - name: resource_name + config: + tags: string + # Optional: Add specific properties for models columns: - name: column_name - tags: [] + tags: string tests: - : + test-name: config: - tags: | [] + tags: string ``` - -```jinja - + +```sql {{ config( tags="" | [""] ) }} - ``` + @@ -191,22 +198,22 @@ seeds: -### Apply tags to exports +### Apply tags to saved queries -Applying tags to exports is only available in dbt Core versions 1.9 and later. +Applying tags to saved queries is only available in dbt Core versions 1.9 and later. -This following example shows how to apply tags to an export in the `dbt_project.yml` file. The export is then tagged with `order_metrics`. +This following example shows how to apply a tag to a saved query in the `dbt_project.yml` file. The saved query is then tagged with `order_metrics`. ```yml -[exports](/docs/build/saved-queries#configure-exports): +[saved-queries](/docs/build/saved-queries): jaffle_shop: customer_order_metrics: +tags: order_metrics @@ -214,23 +221,19 @@ This following example shows how to apply tags to an export in the `dbt_project. -The second example shows how to apply tags to an export in the `semantic_model.yml` file. The export is then tagged with `order_metrics` and `hourly`. +The second example shows how to apply multiple tags to a saved query in the `semantic_model.yml` file. The saved query is then tagged with `order_metrics` and `hourly`. ```yaml saved_queries: - - name: order_metrics - ... - exports: - - name: hourly_order_metrics - config: - alias: my_export_alias - export_as: table - schema: my_export_schema_name - tags: - - order_metrics - - hourly + - name: test_saved_query + description: "{{ doc('saved_query_description') }}" + label: Test saved query + config: + tags: + - order_metrics + - hourly ``` From af4165628afc27b9323cf788d9e059ba0abdccc9 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Fri, 13 Dec 2024 11:39:50 +0000 Subject: [PATCH 17/30] add config --- website/docs/docs/build/dimensions.md | 15 ++++++---- .../docs/reference/resource-configs/meta.md | 29 +++++++++++-------- 2 files changed, 26 insertions(+), 18 deletions(-) diff --git a/website/docs/docs/build/dimensions.md b/website/docs/docs/build/dimensions.md index 4ad06e5626e..25a1c729a7a 100644 --- a/website/docs/docs/build/dimensions.md +++ b/website/docs/docs/build/dimensions.md @@ -62,8 +62,9 @@ semantic_models: type_params: time_granularity: day label: "Date of transaction" # Recommend adding a label to provide more context to users consuming the data - meta: - data_owner: "Finance team" + config: + meta: + data_owner: "Finance team" expr: ts - name: is_bulk type: categorical @@ -147,8 +148,9 @@ dimensions: - name: is_bulk_transaction type: categorical expr: case when quantity > 10 then true else false end - meta: - usage: "Filter to identify bulk transactions, like where quantity > 10." + config: + meta: + usage: "Filter to identify bulk transactions, like where quantity > 10." ``` @@ -192,8 +194,9 @@ dimensions: type: time label: "Date of creation" expr: ts_created # ts_created is the underlying column name from the table - meta: - notes: "Only valid for orders from 2022 onward" + config: + meta: + notes: "Only valid for orders from 2022 onward" is_partition: True type_params: time_granularity: day diff --git a/website/docs/reference/resource-configs/meta.md b/website/docs/reference/resource-configs/meta.md index dcf04354dd6..7214dac560b 100644 --- a/website/docs/reference/resource-configs/meta.md +++ b/website/docs/reference/resource-configs/meta.md @@ -436,21 +436,24 @@ semantic_models: dimensions: - name: order_date type: time - meta: - data_owner: "Finance team" - used_in_reporting: true + config: + meta: + data_owner: "Finance team" + used_in_reporting: true entities: - name: customer_id type: primary - meta: - description: "Unique identifier for customers" - data_owner: "Sales team" - used_in_reporting: false + config: + meta: + description: "Unique identifier for customers" + data_owner: "Sales team" + used_in_reporting: false measures: - name: count_of_users expr: user_id - meta: - used_in_reporting: true + config: + meta: + used_in_reporting: true ``` @@ -468,11 +471,13 @@ semantic-models: ... [dimensions](/docs/build/dimensions): - name: order_date - +meta: - data_owner: "Finance team" - used_in_reporting: true + config: + meta: + data_owner: "Finance team" + used_in_reporting: true ``` + From 8b331eea0b5f4df16f54b4d70454f12d7798de04 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Fri, 20 Dec 2024 15:59:36 +0000 Subject: [PATCH 18/30] add callouts about email mismatch --- .../cloud/manage-access/external-oauth.md | 50 ++++++++++++------- 1 file changed, 31 insertions(+), 19 deletions(-) diff --git a/website/docs/docs/cloud/manage-access/external-oauth.md b/website/docs/docs/cloud/manage-access/external-oauth.md index 380d0a3d1cc..8f434657ad2 100644 --- a/website/docs/docs/cloud/manage-access/external-oauth.md +++ b/website/docs/docs/cloud/manage-access/external-oauth.md @@ -29,6 +29,11 @@ The process of setting up external OAuth will require a little bit of back-and-f If the admins that handle these products are all different people, it’s better to have them coordinating simultaneously to reduce friction. +:::info Username and credential matching required +Ensure that the username entered by the IdP admin matches the username in the Snowflake credentials. If the email address used in the dbt Cloud setup is different from the Snowflake username, the connection will fail or you may run into issues. +::: + + ### Snowflake commands The following is a template for creating the OAuth configurations in the Snowflake environment: @@ -45,18 +50,22 @@ external_oauth_audience_list = ('') external_oauth_token_user_mapping_claim = 'sub' external_oauth_snowflake_user_mapping_attribute = 'email_address' external_oauth_any_role_mode = 'ENABLE' - ``` The `external_oauth_token_user_mapping_claim` and `external_oauth_snowflake_user_mapping_attribute` can be modified based on the your organizations needs. These values point to the claim in the users’ token. In the example, Snowflake will look up the Snowflake user whose `email` matches the value in the `sub` claim. -**Note:** The Snowflake default roles ACCOUNTADMIN, ORGADMIN, or SECURITYADMIN, are blocked from external OAuth by default and they will likely fail to authenticate. See the [Snowflake documentation](https://docs.snowflake.com/en/sql-reference/sql/create-security-integration-oauth-external) for more information. +**Notes:** +- The Snowflake default roles ACCOUNTADMIN, ORGADMIN, or SECURITYADMIN, are blocked from external OAuth by default and they will likely fail to authenticate. See the [Snowflake documentation](https://docs.snowflake.com/en/sql-reference/sql/create-security-integration-oauth-external) for more information. +- The value for `external_oauth_snowflake_user_mapping_attribute` must map correctly to the Snowflake username. For example, if `email_address` is used, the email in the token from the IdP must match the Snowflake username exactly. ## Identity provider configuration -Select a supported identity provider (IdP) for instructions on configuring external OAuth in their environment and completing the integration in dbt Cloud. +Select a supported identity provider (IdP) for instructions on configuring external OAuth in their environment and completing the integration in dbt Cloud: - +- [Okta](#okta) +- [Entra ID](#entra-id) + +## Okta ### 1. Initialize the dbt Cloud settings @@ -139,6 +148,11 @@ Adjust the other settings as needed to meet your organization's configurations i 3. Run the steps to create the integration in Snowflake. +:::info Username consistency +Ensure that the username (for example, email address) entered in the IdP matches the Snowflake credentials for all users. Mismatched usernames will result in authentication failures. +::: + + ### 5. Configuring the integration in dbt Cloud 1. Navigate back to the dbt Cloud **Account settings** —> **Integrations** page you were on at the beginning. It’s time to start filling out all of the fields. @@ -169,9 +183,9 @@ Adjust the other settings as needed to meet your organization's configurations i 4. **Save** the connection, and you have now configured External OAuth with Okta and Snowflake! - - + +## Entra ID ### 1. Initialize the dbt Cloud settings @@ -179,21 +193,19 @@ Adjust the other settings as needed to meet your organization's configurations i 2. Scroll down to **Custom integrations** and click **Add integrations**. 3. Leave this window open. You can set the **Integration type** to Entra ID and note the **Redirect URI** at the bottom of the page. Copy this to your clipboard for use in the next steps. -### Entra ID +### 2. Create the Entra ID apps -You’ll create two apps in the Azure portal: A resource server and a client app. +- You’ll create two apps in the Azure portal: A resource server and a client app. +- In your Azure portal, open the **Entra ID** and click **App registrations** from the left menu. :::important -The admin who creates the apps in the Microsoft Entra ID account must also be a user in Snowflake. - -The `value` field gathered in these steps is only displayed once. When created, record it immediately. - +- The admin who creates the apps in the Microsoft Entra ID account must also be a user in Snowflake. +- The `value` field gathered in these steps is only displayed once. When created, record it immediately. +- Ensure that the username (for example, email address) entered in the IdP matches the Snowflake credentials for all users. Mismatched usernames will result in authentication failures. ::: -In your Azure portal, open the **Entra ID** and click **App registrations** from the left menu. - -### 1. Create a resource server +### 3. Create a resource server 1. From the app registrations screen, click **New registration**. 1. Give the app a name. @@ -209,7 +221,7 @@ In your Azure portal, open the **Entra ID** and click **App registrations** from 4. Ensure **State** is set to **Enabled**. 5. Click **Add scope**. -### 2. Create a client app +### 4. Create a client app 1. From the **App registration page**, click **New registration**. 1. Give the app a name that uniquely identifies it as the client app. @@ -225,7 +237,7 @@ In your Azure portal, open the **Entra ID** and click **App registrations** from 7. Record the `value` for use in a future step and record it immediately. **Note**: Entra ID will not display this value again once you navigate away from this screen. -### 3. Snowflake configuration +### 5. Snowflake configuration You'll be switching between the Entra ID site and Snowflake. Keep your Entra ID account open for this process. @@ -257,7 +269,7 @@ app in Entra ID, click **Endpoints** and open the **Federation metadata document - The **Application ID URI** maps to the `external_oauth_audience_list` field in Snowflake. 4. Run the configurations. Be sure the admin who created the Microsoft apps is also a user in Snowflake, or the configuration will fail. -### 4. Configuring the integration in dbt Cloud +### 6. Configuring the integration in dbt Cloud 1. Navigate back to the dbt Cloud **Account settings** —> **Integrations** page you were on at the beginning. It’s time to start filling out all of the fields. There will be some back-and-forth between the Entra ID account and dbt Cloud. 2. `Integration name`: Give the integration a descriptive name that includes identifying information about the Entra ID environment so future users won’t have to guess where it belongs. @@ -266,7 +278,7 @@ app in Entra ID, click **Endpoints** and open the **Federation metadata document 5. `Authorization URL` and `Token URL`: From the client ID app, open the `Endpoints` tab. These URLs map to the `OAuth 2.0 authorization endpoint (v2)` and `OAuth 2.0 token endpoint (v2)` fields. *You must use v2 of the `OAuth 2.0 authorization endpoint`. Do not use V1.* You can use either version of the `OAuth 2.0 token endpoint`. 6. `Application ID URI`: Copy the `Application ID URI` field from the resource server’s Overview screen. - + ## FAQs From 3cd8a0e7fcdd1dac24b193198917a54222bbb24c Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 23 Dec 2024 10:10:01 +0000 Subject: [PATCH 19/30] add explorer udpates this pr updates the explorer doc so it includes the recent changes: - support for saved queries - support for 'contains' logic in search --- website/docs/docs/collaborate/explore-projects.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/docs/docs/collaborate/explore-projects.md b/website/docs/docs/collaborate/explore-projects.md index 0e4dd7963c8..4068149187d 100644 --- a/website/docs/docs/collaborate/explore-projects.md +++ b/website/docs/docs/collaborate/explore-projects.md @@ -113,7 +113,7 @@ Lenses are helpful to analyze a subset of the DAG if you're zoomed in, or to fin A resource in your project is characterized by resource type, materialization type, or model layer, as well as its latest run or latest test status. Lenses are available for the following metadata: -- **Relationship**: Organizes resources by resource type, such as models, tests, seeds, and [more](/reference/node-selection/syntax). Resource type uses the `resource_type` selector. +- **Resource type**: Organizes resources by resource type, such as models, tests, seeds, saved query, and [more](/docs/build/projects). Resource type uses the `resource_type` selector. - **Materialization type**: Identifies the strategy for building the dbt models in your data platform. - **Latest status**: The status from the latest execution of the resource in the current environment. For example, diagnosing a failed DAG region. - **Model layer**: The modeling layer that the model belongs to according to [best practices guide](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview#guide-structure-overview). For example, discovering marts models to analyze. @@ -121,13 +121,13 @@ A resource in your project is characterized by resource type, materialization ty - **Intermediate** — A model with the prefix `int_`. Or, a model that lives in the `/int/` or `/intermediate/` subdirectory. - **Staging** — A model with the prefix `stg_`. Or, a model that lives in the `/staging/` subdirectory. - **Test status**: The status from the latest execution of the tests that ran again this resource. In the case that a model has multiple tests with different results, the lens reflects the 'worst case' status. -- **Usage queries**: The number of queries against this resource over a given time period. +- **Consumption query history**: The number of queries against this resource over a given time period. ### Example of lenses -Example of applying the **Materialization Type** _lens_ with the lineage graph zoomed out. In this view, each model name has a color according to the materialization type legend at the bottom, which specifies the materialization type. This color-coding helps to quickly identify the materialization types of different models. +Example of applying the **Materialization type** _lens_ with the lineage graph zoomed out. In this view, each model name has a color according to the materialization type legend at the bottom, which specifies the materialization type. This color-coding helps to quickly identify the materialization types of different models. @@ -141,7 +141,7 @@ You can locate resources in your project by performing a keyword search in the s -- **Partial keyword search** — This is also referred to as fuzzy search. +- **Partial keyword search** — Also referred to as fuzzy search. Explorer also uses a "contains" logic to improve your search results. This means you don't need to know the exact root word of your search term. - **Exclude keywords** — Prepend a minus sign (-) to the keyword you want to exclude from search results. For example, `-user` will exclude all matches of that keyword from search results. - **Boolean operators** — Use Boolean operators to enhance your keyword search. For example, the search results for `users OR github` will include matches for either keyword. - **Phrase search** — Surround a string of keywords with double quotation marks to search for that exact phrase (for example, `"stg users"`). To learn more, refer to [Phrase search](https://en.wikipedia.org/wiki/Phrase_search) on Wikipedia. From 952374035b02b072216dfefcc7bb499b5edbb74e Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Mon, 23 Dec 2024 12:07:35 +0000 Subject: [PATCH 20/30] feedback --- website/docs/reference/resource-configs/tags.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index 45c4e719059..7e0016f0d35 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -22,7 +22,7 @@ datatype: string | [string] [models](/reference/model-configs): [](/reference/resource-configs/resource-path): - +tags: | [] + +tags: | [] # Supports single strings or list of strings [snapshots](/reference/snapshot-configs): [](/reference/resource-configs/resource-path): @@ -41,7 +41,7 @@ datatype: string | [string] [models](/reference/model-configs): [](/reference/resource-configs/resource-path): - +tags: | [] + +tags: | [] # Supports single strings or list of strings [snapshots](/reference/snapshot-configs): [](/reference/resource-configs/resource-path): @@ -80,15 +80,16 @@ The following examples show how to add tags to dbt resources in YAML files. Repl resource_type: - name: resource_name config: - tags: string - # Optional: Add specific properties for models + tags: | [] # Supports single strings or list of strings + # Optional: Add the following specific properties for models columns: - name: column_name - tags: string + tags: | [] tests: test-name: config: - tags: string + tags: "single-string" # Supports single string + tags: ["string-1", "string-2"] # Supports list of strings ``` From 90d624caa2632f9ba7fa40eb7cd8e32589b45217 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 23 Dec 2024 16:20:20 +0000 Subject: [PATCH 21/30] Update website/docs/docs/collaborate/explore-projects.md --- website/docs/docs/collaborate/explore-projects.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/collaborate/explore-projects.md b/website/docs/docs/collaborate/explore-projects.md index 4068149187d..fb16c89ef39 100644 --- a/website/docs/docs/collaborate/explore-projects.md +++ b/website/docs/docs/collaborate/explore-projects.md @@ -141,7 +141,7 @@ You can locate resources in your project by performing a keyword search in the s -- **Partial keyword search** — Also referred to as fuzzy search. Explorer also uses a "contains" logic to improve your search results. This means you don't need to know the exact root word of your search term. +- **Partial keyword search** — Also referred to as fuzzy search. Explorer uses a "contains" logic to improve your search results. This means you don't need to know the exact root word of your search term. - **Exclude keywords** — Prepend a minus sign (-) to the keyword you want to exclude from search results. For example, `-user` will exclude all matches of that keyword from search results. - **Boolean operators** — Use Boolean operators to enhance your keyword search. For example, the search results for `users OR github` will include matches for either keyword. - **Phrase search** — Surround a string of keywords with double quotation marks to search for that exact phrase (for example, `"stg users"`). To learn more, refer to [Phrase search](https://en.wikipedia.org/wiki/Phrase_search) on Wikipedia. From b214a2411727e921fe512c57f93461d1d6b62a37 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 23 Dec 2024 22:05:57 +0000 Subject: [PATCH 22/30] clarify full-refresh behavior (#6700) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit this pr clarifies full_refresh config behavior when you set it to `false` and `true`. Resolves #6698 --- 🚀 Deployment available! Here are the direct links to the updated files: - https://docs-getdbt-com-git-update-full-refresh-dbt-labs.vercel.app/docs/build/incremental-microbatch - https://docs-getdbt-com-git-update-full-refresh-dbt-labs.vercel.app/reference/resource-configs/full_refresh --------- Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com> --- .../docs/docs/build/incremental-microbatch.md | 2 +- .../resource-configs/full_refresh.md | 35 +++++++++++-------- 2 files changed, 21 insertions(+), 16 deletions(-) diff --git a/website/docs/docs/build/incremental-microbatch.md b/website/docs/docs/build/incremental-microbatch.md index 4aff8b5839c..77f9ac274b0 100644 --- a/website/docs/docs/build/incremental-microbatch.md +++ b/website/docs/docs/build/incremental-microbatch.md @@ -232,7 +232,7 @@ from {{ source('sales', 'transactions') }} ### Full refresh -As a best practice, we recommend configuring `full_refresh: False` on microbatch models so that they ignore invocations with the `--full-refresh` flag. If you need to reprocess historical data, do so with a targeted backfill that specifies explicit start and end dates. +As a best practice, we recommend [configuring `full_refresh: false`](/reference/resource-configs/full_refresh) on microbatch models so that they ignore invocations with the `--full-refresh` flag. If you need to reprocess historical data, do so with a targeted backfill that specifies explicit start and end dates. ## Usage diff --git a/website/docs/reference/resource-configs/full_refresh.md b/website/docs/reference/resource-configs/full_refresh.md index c874fe7a396..5e291fa2454 100644 --- a/website/docs/reference/resource-configs/full_refresh.md +++ b/website/docs/reference/resource-configs/full_refresh.md @@ -4,6 +4,8 @@ description: "Setting the full_refresh config to false prevents a model or seed datatype: boolean --- +The `full_refresh` config allows you to control whether a resource will always or never perform a full-refresh. This config overrides the `--full-refresh` command-line flag. + ](/reference/resource-configs/resource-path): - +full_refresh: false - + +full_refresh: false | true ``` @@ -30,17 +31,14 @@ models: ```sql {{ config( - full_refresh = false + full_refresh = false | true ) }} select ... - ``` -The configured model(s) will not full-refresh when `dbt run --full-refresh` is invoked. - @@ -50,27 +48,34 @@ The configured model(s) will not full-refresh when `dbt run --full-refresh` is i ```yml seeds: [](/reference/resource-configs/resource-path): - +full_refresh: false + +full_refresh: false | true ``` -The configured seed(s) will not full-refresh when `dbt seed --full-refresh` is invoked. - +- If `full_refresh:true` — the configured resources(s) will full-refresh when `dbt run --full-refresh` is invoked. +- If `full_refresh:false` — the configured resources(s) will _not_ full-refresh when `dbt run --full-refresh` is invoked. + + ## Description -Optionally set a resource to always or never full-refresh. -- If specified as `true` or `false`, the -`full_refresh` config will take precedence over the presence or absence of the `--full-refresh` flag. -- If the `full_refresh` config is `none` or omitted, the resource will use the value of the `--full-refresh` flag. -**Note:** The `--full-refresh` flag also supports a short name, `-f`. +The `full_refresh` config allows you to optionally configure whether a resource will always or never perform a full-refresh. This config is an override for the `--full-refresh` command line flag used when running dbt commands. + + +| `full_refresh` value | Behavior | +| ---------------------------- | -------- | +| `true` | The resource always full-refreshes, regardless of the presence or absence of the `--full-refresh` flag. | +| `false` | The resource never full-refreshes, even if the `--full-refresh` flag is provided. | +| `none` or omitted | The resource follows the behavior of the `--full-refresh` flag. If the flag is used, the resource will full-refresh; otherwise, it won't. | -This logic is encoded in the [`should_full_refresh()`](https://github.com/dbt-labs/dbt-adapters/blob/60005a0a2bd33b61cb65a591bc1604b1b3fd25d5/dbt/include/global_project/macros/materializations/configs.sql) macro. +#### Note +- The `--full-refresh` flag also supports a short name, `-f`. +- The [`should_full_refresh()`](https://github.com/dbt-labs/dbt-adapters/blob/60005a0a2bd33b61cb65a591bc1604b1b3fd25d5/dbt/include/global_project/macros/materializations/configs.sql) macro has logic encoded. ## Usage From 7224646ffea749b7472a7d21bfcf3018f12ddbe5 Mon Sep 17 00:00:00 2001 From: "Leona B. Campbell" <3880403+runleonarun@users.noreply.github.com> Date: Mon, 23 Dec 2024 16:31:18 -0800 Subject: [PATCH 23/30] Update website/docs/docs/collaborate/explore-projects.md --- website/docs/docs/collaborate/explore-projects.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/collaborate/explore-projects.md b/website/docs/docs/collaborate/explore-projects.md index fb16c89ef39..627b255cd78 100644 --- a/website/docs/docs/collaborate/explore-projects.md +++ b/website/docs/docs/collaborate/explore-projects.md @@ -141,7 +141,7 @@ You can locate resources in your project by performing a keyword search in the s -- **Partial keyword search** — Also referred to as fuzzy search. Explorer uses a "contains" logic to improve your search results. This means you don't need to know the exact root word of your search term. +- **Partial keyword search** — Also referred to as fuzzy search. Explorer uses a "contains" logic to improve your search results. This means you can search for partial terms without knowing the exact root word of your search term. - **Exclude keywords** — Prepend a minus sign (-) to the keyword you want to exclude from search results. For example, `-user` will exclude all matches of that keyword from search results. - **Boolean operators** — Use Boolean operators to enhance your keyword search. For example, the search results for `users OR github` will include matches for either keyword. - **Phrase search** — Surround a string of keywords with double quotation marks to search for that exact phrase (for example, `"stg users"`). To learn more, refer to [Phrase search](https://en.wikipedia.org/wiki/Phrase_search) on Wikipedia. From cf53465ec65837849472a948dd2e92c20cd131dc Mon Sep 17 00:00:00 2001 From: Anks S <6273915+asarraf@users.noreply.github.com> Date: Mon, 23 Dec 2024 20:58:11 -0800 Subject: [PATCH 24/30] Update 08-upgrading-to-v1.7.md Correct the instructions for `dbt clean` section of Upgrading to 1.7 --- .../docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.7.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.7.md b/website/docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.7.md index df24b63a2f0..b98a76295cf 100644 --- a/website/docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.7.md +++ b/website/docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.7.md @@ -66,7 +66,7 @@ dbt Core v1.5 introduced model governance which we're continuing to refine. v1. ### dbt clean -Starting in v1.7, `dbt clean` will only clean paths within the current working directory. The `--no-clean-project-files-only` flag will delete all paths specified in `clean-paths`, even if they're outside the dbt project. +Starting in v1.7, `dbt clean` will only clean paths within the current working directory. The `--no-clean-project-files-only` flag will delete all paths specified in the `clean-targets` section of `dbt_project.yml`, even if they're outside the dbt project. Supported flags: - `--clean-project-files-only` (default) From 26e35aff3a4dda83a69071f90fb8899ceee1e84e Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Tue, 24 Dec 2024 09:37:22 +0000 Subject: [PATCH 25/30] update title --- website/docs/docs/cloud/manage-access/external-oauth.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/website/docs/docs/cloud/manage-access/external-oauth.md b/website/docs/docs/cloud/manage-access/external-oauth.md index 121ba8f4cb4..ccb8fd66cc3 100644 --- a/website/docs/docs/cloud/manage-access/external-oauth.md +++ b/website/docs/docs/cloud/manage-access/external-oauth.md @@ -22,18 +22,17 @@ dbt Cloud Enterprise supports [external OAuth authentication](https://docs.snow The process of setting up external OAuth will require a little bit of back-and-forth between your dbt Cloud, IdP, and Snowflake accounts, and having them open in multiple browser tabs will help speed up the configuration process: -- **dbt Cloud:** You’ll primarily be working in the **Account Settings** —> **Integrations** page. You will need [proper permission](/docs/cloud/manage-access/enterprise-permissions) to set up the integration and create the connections. +- **dbt Cloud:** You’ll primarily be working in the **Account settings** —> **Integrations** page. You will need [proper permission](/docs/cloud/manage-access/enterprise-permissions) to set up the integration and create the connections. - **Snowflake:** Open a worksheet in an account that has permissions to [create a security integration](https://docs.snowflake.com/en/sql-reference/sql/create-security-integration). - **Okta:** You’ll be working in multiple areas of the Okta account, but you can start in the **Applications** section. You will need permissions to [create an application](https://help.okta.com/en-us/content/topics/security/custom-admin-role/about-role-permissions.htm#Application_permissions) and an [authorization server](https://help.okta.com/en-us/content/topics/security/custom-admin-role/about-role-permissions.htm#Authorization_server_permissions). - **Entra ID** An admin with access to create [Entra ID apps](https://learn.microsoft.com/en-us/entra/identity/role-based-access-control/custom-available-permissions) who is also a user in Snowflake is required. If the admins that handle these products are all different people, it’s better to have them coordinating simultaneously to reduce friction. -:::info Username and credential matching required -Ensure that the username entered by the IdP admin matches the username in the Snowflake credentials. If the email address used in the dbt Cloud setup is different from the Snowflake username, the connection will fail or you may run into issues. +:::info Snowflake and IdP username matching required +Ensure that the username/email address entered by the IdP admin matches the Snowflake credentials username. If the email address used in the dbt Cloud setup is different from the Snowflake email address, the connection will fail or you may run into issues. ::: - ### Snowflake commands The following is a template for creating the OAuth configurations in the Snowflake environment: From 5fc9357903e14a525edc9c1a4f076f5090203f33 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Tue, 24 Dec 2024 10:42:29 +0000 Subject: [PATCH 26/30] clarify meta acceptance --- website/docs/reference/resource-configs/meta.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/meta.md b/website/docs/reference/resource-configs/meta.md index 7214dac560b..313595ea497 100644 --- a/website/docs/reference/resource-configs/meta.md +++ b/website/docs/reference/resource-configs/meta.md @@ -290,7 +290,7 @@ saved_queries: ## Definition -The `meta` field can be used to set metadata for a resource. This metadata is compiled into the `manifest.json` file generated by dbt, and is viewable in the auto-generated documentation. +The `meta` field can be used to set metadata for a resource and accepts any key-value pairs. This metadata is compiled into the `manifest.json` file generated by dbt, and is viewable in the auto-generated documentation. Depending on the resource you're configuring, `meta` may be available within the `config` property, and/or as a top-level key. (For backwards compatibility, `meta` is often (but not always) supported as a top-level key, though without the capabilities of config inheritance.) From 3e561f12b0b1e35a531e0c17363218d3beadf64b Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 24 Dec 2024 10:53:36 +0000 Subject: [PATCH 27/30] Update tags.md add command examples --- website/docs/reference/resource-configs/tags.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index 7e0016f0d35..27be1f71766 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -114,6 +114,7 @@ Apply a tag (or list of tags) to a resource. These tags can be used as part of the [resource selection syntax](/reference/node-selection/syntax), when running the following commands: - `dbt run --select tag:my_tag` +- `dbt build --select tag:my_tag` - `dbt seed --select tag:my_tag` - `dbt snapshot --select tag:my_tag` - `dbt test --select tag:my_tag` (indirectly runs all tests associated with the models that are tagged) @@ -222,6 +223,13 @@ This following example shows how to apply a tag to a saved query in the `dbt_pro +Then, run part of your project like so: + +``` +# Run all resources tagged "order_metrics" +dbt run --select tag:order_metrics +``` + The second example shows how to apply multiple tags to a saved query in the `semantic_model.yml` file. The saved query is then tagged with `order_metrics` and `hourly`. @@ -236,8 +244,15 @@ saved_queries: - order_metrics - hourly ``` - + + +Then, run part of your project like so: + +``` +# Run all resources tagged "order_metrics" and "hourly" +dbt build --select tag:order_metrics tag:hourly +``` ## Usage notes From 41a5d19b6655834424d02e4b23921ac1c607a975 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 24 Dec 2024 10:57:30 +0000 Subject: [PATCH 28/30] Update meta.md add link --- website/docs/reference/resource-configs/meta.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/reference/resource-configs/meta.md b/website/docs/reference/resource-configs/meta.md index 313595ea497..a7f348d50ba 100644 --- a/website/docs/reference/resource-configs/meta.md +++ b/website/docs/reference/resource-configs/meta.md @@ -179,7 +179,7 @@ exposures: -Configure `meta` in the your semantic models YAML file or under the `semantic-models` config block in the `dbt_project.yml` file. +Configure `meta` in the your [semantic models](/docs/build/semantic-models) YAML file or under the `semantic-models` config block in the `dbt_project.yml` file. @@ -381,7 +381,7 @@ models: ### Assign meta to semantic model -The following example shows how to assign a `meta` value to a semantic model in the `semantic_model.yml` file and `dbt_project.yml` file: +The following example shows how to assign a `meta` value to a [semantic model](/docs/build/semantic-models) in the `semantic_model.yml` file and `dbt_project.yml` file: From f98a1a70fb5f4947f5cdd177572381c459155457 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Tue, 24 Dec 2024 12:21:45 +0000 Subject: [PATCH 29/30] natalies feedback --- .../docs/reference/resource-configs/tags.md | 31 ++++++++++--------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index 27be1f71766..b77c1472032 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -164,14 +164,14 @@ select ... -Then, run part of your project like so: +Run resources with specific tags (or exclude resources with specific tags) with the following commands: -``` +```shell # Run all models tagged "daily" -$ dbt run --select tag:daily + dbt run --select tag:daily # Run all models tagged "daily", except those that are tagged hourly -$ dbt run --select tag:daily --exclude tag:hourly + dbt run --select tag:daily --exclude tag:hourly ``` ### Apply tags to seeds @@ -204,7 +204,10 @@ seeds: +:::tip Upgrade to dbt Core 1.9 + Applying tags to saved queries is only available in dbt Core versions 1.9 and later. +::: @@ -223,11 +226,11 @@ This following example shows how to apply a tag to a saved query in the `dbt_pro -Then, run part of your project like so: +Then run resources with a specific tag with the following commands: -``` +```shell # Run all resources tagged "order_metrics" -dbt run --select tag:order_metrics + dbt run --select tag:order_metrics ``` The second example shows how to apply multiple tags to a saved query in the `semantic_model.yml` file. The saved query is then tagged with `order_metrics` and `hourly`. @@ -247,11 +250,11 @@ saved_queries: -Then, run part of your project like so: +Run resources with multiple tags with the following commands: -``` +```shell # Run all resources tagged "order_metrics" and "hourly" -dbt build --select tag:order_metrics tag:hourly + dbt build --select tag:order_metrics tag:hourly ``` @@ -304,8 +307,8 @@ sources: In the example above, the `unique` test would be selected by any of these four tags: ```bash -$ dbt test --select tag:top_level -$ dbt test --select tag:table_level -$ dbt test --select tag:column_level -$ dbt test --select tag:test_level +dbt test --select tag:top_level +dbt test --select tag:table_level +dbt test --select tag:column_level +dbt test --select tag:test_level ``` From 642912c84934c807d3cbaf19f077094b48fcecaf Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Tue, 24 Dec 2024 12:22:57 +0000 Subject: [PATCH 30/30] grammar fix --- website/docs/reference/resource-configs/tags.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index b77c1472032..c222df8c1ae 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -164,7 +164,7 @@ select ... -Run resources with specific tags (or exclude resources with specific tags) with the following commands: +Run resources with specific tags (or exclude resources with specific tags) using the following commands: ```shell # Run all models tagged "daily" @@ -226,7 +226,7 @@ This following example shows how to apply a tag to a saved query in the `dbt_pro -Then run resources with a specific tag with the following commands: +Then run resources with a specific tag using the following commands: ```shell # Run all resources tagged "order_metrics" @@ -250,7 +250,7 @@ saved_queries: -Run resources with multiple tags with the following commands: +Run resources with multiple tags using the following commands: ```shell # Run all resources tagged "order_metrics" and "hourly"