From 2ccbc79b386aa9193357cb0ccbf643f4f55a1f19 Mon Sep 17 00:00:00 2001 From: Mike Alfare Date: Mon, 4 Dec 2023 17:27:58 -0500 Subject: [PATCH 1/7] add dynamic table configuration for snowflake --- .../resource-configs/snowflake-configs.md | 85 ++++++++++--------- 1 file changed, 46 insertions(+), 39 deletions(-) diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md index 30c7966ec68..6c7ba86d353 100644 --- a/website/docs/reference/resource-configs/snowflake-configs.md +++ b/website/docs/reference/resource-configs/snowflake-configs.md @@ -346,82 +346,87 @@ In the configuration format for the model SQL file: ## Dynamic tables -The Snowflake adapter supports [dynamic tables](https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table). +The Snowflake adapter supports [dynamic tables](https://docs.snowflake.com/en/user-guide/dynamic-tables-about). This materialization is specific to Snowflake, which means that any model configuration that would normally come along for the ride from `dbt-core` (e.g. as with a `view`) may not be available for dynamic tables. This gap will decrease in future patches and versions. While this materialization is specific to Snowflake, it very much follows the implementation of [materialized views](/docs/build/materializations#Materialized-View). In particular, dynamic tables have access to the `on_configuration_change` setting. -There are also some limitations that we hope to address in the next version. +Dynamic tables are supported with the following configuration parameters: -### Parameters +| Parameter | Type | Required | Default | Change Monitoring Support | Reference | +|---------------------------|----------|----------|-----------|---------------------------|----------------------------------------------| +| `on_configuration_change` | STRING | NO | `'apply'` | N/A | | +| `target_lag` | STRING | YES | | ALTER | [Target lag](#target-lag) | +| `snowflake_warehouse` | STRING | YES | | ALTER | [Warehouse](#configuring-virtual-warehouses) | -Dynamic tables in `dbt-snowflake` require the following parameters: -- `target_lag` -- `snowflake_warehouse` -- `on_configuration_change` +#### Sample model file: -To learn more about each parameter and what values it can take, see -the Snowflake docs page: [`CREATE DYNAMIC TABLE: Parameters`](https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table). - -### Usage - -You can create a dynamic table by editing _one_ of these files: - -- the SQL file for your model -- the `dbt_project.yml` configuration file - -The following examples create a dynamic table: - - + ```sql {{ config( - materialized = 'dynamic_table', - snowflake_warehouse = 'snowflake_warehouse', - target_lag = '10 minutes', + materialized='dynamic_table', + on_configuration_change='', + target_lag='< { seconds | minutes | hours | days } | DOWNSTREAM>', + snowflake_warehouse='', ) }} + +select * from {{ ref('my_base_table') }} ``` +#### Sample project file: + ```yaml models: path: materialized: dynamic_table - snowflake_warehouse: snowflake_warehouse - target_lag: '10 minutes' + on_configuration_change: '' + snowflake_warehouse: '' + target_lag: '< { seconds | minutes | hours | days } | DOWNSTREAM>' ``` -### Monitored configuration changes +Find more information about these parameters in the Snowflake docs: +- [CREATE DYNAMIC TABLE](https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table) -The settings below are monitored for changes applicable to `on_configuration_change`. +### Target lag -#### Target lag +Snowflake allows two configuration scenarios for scheduling automatic refreshes: time-based and dependency-based. +For time-based automatic refreshes, provide a value of the form ` { seconds | minutes | hours | days }`. +For example, if the dynamic table needs to be updated every 30 minutes, use `target_lag='30 minutes'`. +However, for scenarios where the referenced objects are themselves dynamic tables, it might be desirable to refresh +the dynamic table whenever the underlying dynamic table is refreshed. In this scenario, use `target_lag='downstream'`. +This allows for refresh schedules to be controlled once, at the source, instead of at each layer. -Changes to `target_lag` can be applied by running an `ALTER` statement. Refreshing is essentially -always on for dynamic tables; this setting changes how frequently the dynamic table is updated. +### Limitations -#### Warehouse +As with materialized views on most data platforms, there are limitations associated with dynamic tables. Some worth noting include: -Changes to `snowflake_warehouse` can be applied via an `ALTER` statement. +- Dynamic table SQL has a [limited feature set](https://docs.snowflake.com/en/user-guide/dynamic-tables-tasks-create#query-constructs-not-currently-supported-in-dynamic-tables) +- Dynamic table SQL cannot be updated; the dynamic table must go through a `--full-refresh` (DROP/CREATE) +- Dynamic tables cannot be downstream from: materialized views, external tables, streams +- Dynamic tables cannot reference a view that is downstream from another dynamic table -### Limitations +Find more information about dynamic table limitations in Snowflake's [docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-tasks-create#dynamic-table-limitations-and-supported-functions). + + #### Changing materialization to and from "dynamic_table" -Swapping an already materialized model to be a dynamic table and vice versa. -The workaround is manually dropping the existing materialization in the data warehouse prior to calling `dbt run`. -Normally, re-running with the `--full-refresh` flag would resolve this, but not in this case. -This would only need to be done once as the existing object would then be a dynamic table. +Version `1.6.x` does not support altering the materialization from a non-dynamic table be a dynamic table and vice versa. +Re-running with the `--full-refresh` does not resolve this either. +The workaround is manually dropping the existing model in the warehouse prior to calling `dbt run`. +This only needs to be done once for the conversion. For example, assume for the example model below, `my_model`, has already been materialized to the underlying data platform via `dbt run`. -If the user changes the model's config to `materialized="dynamic_table"`, they will get an error. +If the model config is updated to `materialized="dynamic_table"`, dbt will return an error. The workaround is to execute `DROP TABLE my_model` on the data warehouse before trying the model again. @@ -429,7 +434,7 @@ The workaround is to execute `DROP TABLE my_model` on the data warehouse before ```yaml {{ config( - materialized="table" # or any model type eg view, incremental + materialized="table" # or any model type (e.g. view, incremental) ) }} ``` @@ -437,3 +442,5 @@ The workaround is to execute `DROP TABLE my_model` on the data warehouse before + + From 3ca806c8e56a14ebf3c470a19bf625b8342a8379 Mon Sep 17 00:00:00 2001 From: Mike Alfare Date: Tue, 5 Dec 2023 13:25:57 -0500 Subject: [PATCH 2/7] add on_configuration_change --- website/docs/reference/resource-configs/snowflake-configs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md index 6c7ba86d353..39dd4430c4c 100644 --- a/website/docs/reference/resource-configs/snowflake-configs.md +++ b/website/docs/reference/resource-configs/snowflake-configs.md @@ -368,7 +368,7 @@ Dynamic tables are supported with the following configuration parameters: ```sql {{ config( materialized='dynamic_table', - on_configuration_change='', + on_configuration_change='{ apply | continue | fail }', target_lag='< { seconds | minutes | hours | days } | DOWNSTREAM>', snowflake_warehouse='', ) }} From a6f69eb551ce902cef2a9191c3bef052e98e664d Mon Sep 17 00:00:00 2001 From: Mike Alfare <13974384+mikealfare@users.noreply.github.com> Date: Wed, 6 Dec 2023 17:47:36 -0500 Subject: [PATCH 3/7] Update website/docs/reference/resource-configs/snowflake-configs.md Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com> --- website/docs/reference/resource-configs/snowflake-configs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md index 39dd4430c4c..900860ea287 100644 --- a/website/docs/reference/resource-configs/snowflake-configs.md +++ b/website/docs/reference/resource-configs/snowflake-configs.md @@ -357,7 +357,7 @@ Dynamic tables are supported with the following configuration parameters: | Parameter | Type | Required | Default | Change Monitoring Support | Reference | |---------------------------|----------|----------|-----------|---------------------------|----------------------------------------------| -| `on_configuration_change` | STRING | NO | `'apply'` | N/A | | +| `on_configuration_change` | STRING | NO | `apply` | N/A | | | `target_lag` | STRING | YES | | ALTER | [Target lag](#target-lag) | | `snowflake_warehouse` | STRING | YES | | ALTER | [Warehouse](#configuring-virtual-warehouses) | From 503253ef6a6b3ba5f6eb0586e6053bc26dcade6c Mon Sep 17 00:00:00 2001 From: Mike Alfare Date: Thu, 7 Dec 2023 12:33:30 -0500 Subject: [PATCH 4/7] update to use appropriate style guidelines --- .../resource-configs/snowflake-configs.md | 81 ++++++++++++++----- 1 file changed, 59 insertions(+), 22 deletions(-) diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md index 900860ea287..c6ef5c4b339 100644 --- a/website/docs/reference/resource-configs/snowflake-configs.md +++ b/website/docs/reference/resource-configs/snowflake-configs.md @@ -355,44 +355,81 @@ of [materialized views](/docs/build/materializations#Materialized-View). In particular, dynamic tables have access to the `on_configuration_change` setting. Dynamic tables are supported with the following configuration parameters: -| Parameter | Type | Required | Default | Change Monitoring Support | Reference | -|---------------------------|----------|----------|-----------|---------------------------|----------------------------------------------| -| `on_configuration_change` | STRING | NO | `apply` | N/A | | -| `target_lag` | STRING | YES | | ALTER | [Target lag](#target-lag) | -| `snowflake_warehouse` | STRING | YES | | ALTER | [Warehouse](#configuring-virtual-warehouses) | +| Parameter | Type | Required | Default | Change Monitoring Support | +|----------------------------------------------------------|------------|----------|---------|---------------------------| +| `on_configuration_change` | `` | no | `apply` | n/a | +| [`target_lag`](#target-lag) | `` | yes | | alter | +| [`snowflake_warehouse`](#configuring-virtual-warehouses) | `` | yes | | alter | -#### Sample model file: + - -```sql -{{ config( - materialized='dynamic_table', - on_configuration_change='{ apply | continue | fail }', - target_lag='< { seconds | minutes | hours | days } | DOWNSTREAM>', - snowflake_warehouse='', -) }} + + + -select * from {{ ref('my_base_table') }} +```yaml +models: + [](/reference/resource-configs/resource-path): + [+](/reference/resource-configs/plus-prefix)[materialized](/reference/resource-configs/materialized): dynamic_table + [+](/reference/resource-configs/plus-prefix)on_configuration_change: apply | continue | fail + [+](/reference/resource-configs/plus-prefix)[target_lag](#target-lag): downstream | + [+](/reference/resource-configs/plus-prefix)[snowflake_warehouse](#configuring-virtual-warehouses): ``` -#### Sample project file: + - + + + + ```yaml +version: 2 + models: - path: - materialized: dynamic_table - on_configuration_change: '' - snowflake_warehouse: '' - target_lag: '< { seconds | minutes | hours | days } | DOWNSTREAM>' + - name: [] + config: + [materialized](/reference/resource-configs/materialized): dynamic_table + on_configuration_change: apply | continue | fail + [target_lag](#target-lag): downstream | + [snowflake_warehouse](#configuring-virtual-warehouses): ``` + + + + + + + +```jinja +{{ config( + materialized="dynamic_table", + on_configuration_change="apply" | "continue" | "fail", + target_lag="downstream" | " seconds | minutes | hours | days", + snowflake_warehouse="", +) }} +``` + + + + + + + Find more information about these parameters in the Snowflake docs: - [CREATE DYNAMIC TABLE](https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table) From ecc39ab9c905e9750016e7f9d689aeafd508fbbc Mon Sep 17 00:00:00 2001 From: Mike Alfare Date: Fri, 8 Dec 2023 15:35:39 -0500 Subject: [PATCH 5/7] add reference links to config block version --- .../docs/reference/resource-configs/snowflake-configs.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md index c6ef5c4b339..a5c2e6f57d9 100644 --- a/website/docs/reference/resource-configs/snowflake-configs.md +++ b/website/docs/reference/resource-configs/snowflake-configs.md @@ -417,10 +417,10 @@ models: ```jinja {{ config( - materialized="dynamic_table", + [materialized](/reference/resource-configs/materialized)="dynamic_table", on_configuration_change="apply" | "continue" | "fail", - target_lag="downstream" | " seconds | minutes | hours | days", - snowflake_warehouse="", + [target_lag](#target-lag)="downstream" | " seconds | minutes | hours | days", + [snowflake_warehouse](#configuring-virtual-warehouses)="", ) }} ``` From 5d2fd560124e5cf89797a484f52abe3307da736d Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 13 Dec 2023 13:16:18 -0500 Subject: [PATCH 6/7] Update snowflake-configs.md --- .../resource-configs/snowflake-configs.md | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md index a5c2e6f57d9..a24b072d39c 100644 --- a/website/docs/reference/resource-configs/snowflake-configs.md +++ b/website/docs/reference/resource-configs/snowflake-configs.md @@ -435,21 +435,18 @@ Find more information about these parameters in the Snowflake docs: ### Target lag -Snowflake allows two configuration scenarios for scheduling automatic refreshes: time-based and dependency-based. -For time-based automatic refreshes, provide a value of the form ` { seconds | minutes | hours | days }`. -For example, if the dynamic table needs to be updated every 30 minutes, use `target_lag='30 minutes'`. -However, for scenarios where the referenced objects are themselves dynamic tables, it might be desirable to refresh -the dynamic table whenever the underlying dynamic table is refreshed. In this scenario, use `target_lag='downstream'`. -This allows for refresh schedules to be controlled once, at the source, instead of at each layer. +Snowflake allows two configuration scenarios for scheduling automatic refreshes: +- **Time-based** — Provide a value of the form ` { seconds | minutes | hours | days }`. For example, if the dynamic table needs to be updated every 30 minutes, use `target_lag='30 minutes'`. +- **Downstream** — For scenarios where the referenced objects are themselves dynamic tables, it might be desirable to refresh the dynamic table whenever the underlying dynamic table is refreshed. In this scenario, use `target_lag='downstream'`. This allows for refresh schedules to be controlled once, at the source, instead of at each layer. ### Limitations As with materialized views on most data platforms, there are limitations associated with dynamic tables. Some worth noting include: -- Dynamic table SQL has a [limited feature set](https://docs.snowflake.com/en/user-guide/dynamic-tables-tasks-create#query-constructs-not-currently-supported-in-dynamic-tables) -- Dynamic table SQL cannot be updated; the dynamic table must go through a `--full-refresh` (DROP/CREATE) -- Dynamic tables cannot be downstream from: materialized views, external tables, streams -- Dynamic tables cannot reference a view that is downstream from another dynamic table +- Dynamic table SQL has a [limited feature set](https://docs.snowflake.com/en/user-guide/dynamic-tables-tasks-create#query-constructs-not-currently-supported-in-dynamic-tables). +- Dynamic table SQL cannot be updated; the dynamic table must go through a `--full-refresh` (DROP/CREATE). +- Dynamic tables cannot be downstream from: materialized views, external tables, streams. +- Dynamic tables cannot reference a view that is downstream from another dynamic table. Find more information about dynamic table limitations in Snowflake's [docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-tasks-create#dynamic-table-limitations-and-supported-functions). From b84b59eba4b990ef6ca4c1c883b658f0a3f34bd7 Mon Sep 17 00:00:00 2001 From: Mike Alfare Date: Wed, 13 Dec 2023 14:11:45 -0500 Subject: [PATCH 7/7] update downstream option for target_lag, fix external docs reference to be consistent --- .../docs/reference/resource-configs/snowflake-configs.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md index a24b072d39c..aa2bf370df6 100644 --- a/website/docs/reference/resource-configs/snowflake-configs.md +++ b/website/docs/reference/resource-configs/snowflake-configs.md @@ -430,14 +430,15 @@ models: -Find more information about these parameters in the Snowflake docs: -- [CREATE DYNAMIC TABLE](https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table) +Find more information about these parameters in Snowflake's [docs](https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table): ### Target lag Snowflake allows two configuration scenarios for scheduling automatic refreshes: - **Time-based** — Provide a value of the form ` { seconds | minutes | hours | days }`. For example, if the dynamic table needs to be updated every 30 minutes, use `target_lag='30 minutes'`. -- **Downstream** — For scenarios where the referenced objects are themselves dynamic tables, it might be desirable to refresh the dynamic table whenever the underlying dynamic table is refreshed. In this scenario, use `target_lag='downstream'`. This allows for refresh schedules to be controlled once, at the source, instead of at each layer. +- **Downstream** — Applicable when the dynamic table is referenced by other dynamic tables. In this scenario, `target_lag='downstream'` allows for refreshes to be controlled at the target, instead of at each layer. + +Find more information about `target_lag` in Snowflake's [docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh#understanding-target-lag). ### Limitations