diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md index abb516d2258..fb92e9eaf52 100644 --- a/website/docs/reference/resource-configs/snowflake-configs.md +++ b/website/docs/reference/resource-configs/snowflake-configs.md @@ -13,13 +13,15 @@ To-do: ## Iceberg table format -The dbt-snowflake adapter supports the Iceberg table format. It is available for three of the Snowflake materializations: +The dbt-snowflake adapter supports the Iceberg table format. Prior to developing an Iceberg table with dbt, you need to [configure an external volume in Snowflake](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-external-volume). + +In dbt, Iceberg is available for three of the Snowflake materializations: - [Table](/docs/build/materializations#table) - [Incremental](/docs/build/materializations#incremental) -- [Dynamic](#dynamic-tables) +- [Dynamic](#dynamic-tables) -For now, to create Iceberg tables, you must implement a [behavior flag](/reference/global-configs/behavior-changes) due to performance impact related to using Iceberg tables. Snowflake does not support `is_iceberg` on the `Show Objects` query, which dbt depends on for metadata. +For now, to create Iceberg tables, you must implement a [behavior flag](/reference/global-configs/behavior-changes) due to performance impact related to using Iceberg tables. To use Iceberg, set the `enable_iceberg_materializations` flag to `True` in your dbt_project.yml: @@ -34,15 +36,14 @@ flags: - The following configurations are supported. For more information, check out the Snowflake reference for [`CREATE ICEBERG TABLE` (Snowflake as the catalog)](https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-snowflake). -| Field | Type | Required | Description | Sample input | Note | +| Parameter | Type | Required | Description | Sample input | Note | | --------------------- | ------ | -------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Table Format | String | Yes | Configures the objects table format. | `iceberg` | `iceberg` is the only accepted value. | -| External volume | String | Yes(*) | Specifies the identifier (name) of the external volume where Snowflake writes the Iceberg table's metadata and data files. | `my_s3_bucket` | *You don't need to specify this if the account, database, or schema already has an associated external volume. [More info](https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-snowflake#:~:text=Snowflake%20Table%20Structures.-,external_volume) | -| Base location Subpath | String | No | An optional suffix to add to the `base_location` path that dbt automatically specifies. | `jaffle_marketing_folder` | We recommend that you do not specify this. Modifying this parameter results in a new Iceberg table. See [Base Location](#base-location) for more info. | +| `table_format` | String | Yes | Configures the objects table format. | `iceberg` | `iceberg` is the only accepted value. | +| `external_volume` | String | Yes(*) | Specifies the identifier (name) of the external volume where Snowflake writes the Iceberg table's metadata and data files. | `my_s3_bucket` | *You don't need to specify this if the account, database, or schema already has an associated external volume. [More info](https://docs.snowflake.com/user-guide/tables-iceberg-configure-external-volume#set-a-default-external-volume-at-the-account-database-or-schema-level) | +| `base_location_subpath` | String | No | An optional suffix to add to the `base_location` path that dbt automatically specifies. | `jaffle_marketing_folder` | We recommend that you **do not specify this**. Modifying this parameter results in a new Iceberg table. See [Base Location](#base-location) for more info. | ### Example configuration @@ -54,7 +55,7 @@ To configure an Iceberg table materialization in dbt, refer to the example confi {{ config( - materialized = "table", + materialized="table", table_format="iceberg", external_volume="s3_iceberg_snow", ) @@ -79,14 +80,14 @@ A theoretical (but not recommended) use case is re-using an `EXTERNAL VOLUME` wh #### Rationale -dbt manages `base_location` on behalf of users to enforce best practices. With Snowflake-managed Iceberg format tables, the user owns and maintains the data storage of the tables in an external storage solution (the declared `external volume`). The `base_ location` parameter declares where to write the data within the external volume. The Snowflake Iceberg catalog keeps track of your Iceberg table regardless of where the data lives within the `external volume` declared and the `base_location` provided. However, Snowflake permits passing anything into the `base_location` field, including an empty string, even reusing the same path across multiple tables. This behavior could result in future technical debt because it will limit the ability to: +dbt manages `base_location` on behalf of users to enforce best practices. With Snowflake-managed Iceberg format tables, the user owns and maintains the data storage of the tables in an external storage solution (the declared `external volume`). The `base_location` parameter declares where to write the data within the external volume. The Snowflake Iceberg catalog keeps track of your Iceberg table regardless of where the data lives within the `external volume` declared and the `base_location` provided. However, Snowflake permits passing anything into the `base_location` field, including an empty string, even reusing the same path across multiple tables. This behavior could result in future technical debt because it will limit the ability to: - Navigate the underlying object store (S3/Azure blob) - Read Iceberg tables via an object-store integration - Grant schema-specific access to tables via object store - Use a crawler pointed at the tables within the external volume to build a new catalog with another tool -To maintain best practices, we enforce an input. Currently, we do not support overriding the default `base location` input but will consider it based on user feedback. +To maintain best practices, we enforce an input. Currently, we do not support overriding the default `base_location` input but will consider it based on user feedback. In summary, dbt-snowflake does not support arbitrary definition of `base_location` for Iceberg tables. Instead, dbt, by default, writes your tables within a `_dbt/{SCHEMA_NAME}/{TABLE_NAME}` prefix to ensure easier object-store observability and auditability.