diff --git a/website/docs/docs/core/connect-data-platform/firebolt-setup.md b/website/docs/docs/core/connect-data-platform/firebolt-setup.md index 8fb91dea299..b1695c75b37 100644 --- a/website/docs/docs/core/connect-data-platform/firebolt-setup.md +++ b/website/docs/docs/core/connect-data-platform/firebolt-setup.md @@ -40,16 +40,15 @@ To connect to Firebolt from dbt, you'll need to add a [profile](https://docs.get outputs: : type: firebolt - user: "" - password: "" + client_id: "" + client_secret: "" database: "" engine_name: "" + account_name: "" schema: threads: 1 #optional fields - jar_path: host: "" - account_name: "" ``` @@ -57,30 +56,27 @@ To connect to Firebolt from dbt, you'll need to add a [profile](https://docs.get #### Description of Firebolt Profile Fields -To specify values as environment variables, use the format `{{ env_var('' }}`. For example, `{{ env_var('DATABASE_NAME' }}`. +To specify values as environment variables, use the format `{{ env_var('' }}`. For example, `{{ env_var('DATABASE_NAME' }}`. | Field | Description | |--------------------------|--------------------------------------------------------------------------------------------------------| | `type` | This must be included either in `profiles.yml` or in the `dbt_project.yml` file. Must be set to `firebolt`. | -| `user` | Required. A Firebolt username with adequate permissions to access the specified `engine_name`. | -| `password` | Required. The password associated with the specified `user`. | +| `client_id` | Required. Your [service account](https://docs.firebolt.io/godocs/Guides/managing-your-organization/service-accounts.html) id. | +| `client_secret` | Required. The secret associated with the specified `client_id`. | | `database` | Required. The name of the Firebolt database to connect to. | | `engine_name` | Required in version 0.21.10 and later. Optional in earlier versions. The name (not the URL) of the Firebolt engine to use in the specified `database`. This must be a general purpose read-write engine and the engine must be running. If omitted in earlier versions, the default engine for the specified `database` is used. | +| `account_name` | Required. Specifies the account name under which the specified `database` exists. | | `schema` | Recommended. A string to add as a prefix to the names of generated tables when using the [custom schemas workaround](https://docs.getdbt.com/reference/warehouse-profiles/firebolt-profile#supporting-concurrent-development). | -| `threads` | Required. Must be set to `1`. Multi-threading is not currently supported. | -| `jar_path` | Required only with versions earlier than 0.21.0. Ignored in 0.21.0 and later. The path to your JDBC driver on your local drive. | +| `threads` | Required. Set to higher number to improve performance. | | `host` | Optional. The host name of the connection. For all customers it is `api.app.firebolt.io`, which will be used if omitted. | -| `account_name` | Required if more than one account is associated with the specified `user1`. Specifies the account name (not the account ID) under which the specified `database` exists. If omitted, the default account is assumed. | - + #### Troubleshooting Connections If you encounter issues connecting to Firebolt from dbt, make sure the following criteria are met: -- The engine must be a general-purpose read-write engine, not an analytics engine. -- You must have adequate permissions to access the engine. +- You must have adequate permissions to access the engine and the database. +- Your service account must be attached to a user. - The engine must be running. -- If you're not using the default engine for the database, you must specify an engine name. -- If there is more than one account associated with your credentials, you must specify an account. ## Supporting Concurrent Development diff --git a/website/docs/reference/resource-configs/firebolt-configs.md b/website/docs/reference/resource-configs/firebolt-configs.md index 59adee715ba..58fc0e2a319 100644 --- a/website/docs/reference/resource-configs/firebolt-configs.md +++ b/website/docs/reference/resource-configs/firebolt-configs.md @@ -58,7 +58,7 @@ models: table_type: fact primary_index: [ , ... ] indexes: - - type: aggregating | join + - type: aggregating key_column: [ , ... ] aggregation: [ , ... ] ... @@ -96,10 +96,10 @@ models: | Configuration | Description | |-------------------|-------------------------------------------------------------------------------------------| | `materialized` | How the model will be materialized into Firebolt. Must be `table` to create a fact table. | -| `table_type` | Whether the materialized table will be a [fact or dimension](https://docs.firebolt.io/working-with-tables.html#fact-and-dimension-tables) table. | +| `table_type` | Whether the materialized table will be a [fact or dimension](https://docs.firebolt.io/godocs/Overview/working-with-tables/working-with-tables.html#fact-and-dimension-tables) table. | | `primary_index` | Sets the primary index for the fact table using the inputted list of column names from the model. Required for fact tables. | | `indexes` | A list of aggregating indexes to create on the fact table. | -| `type` | Specifies whether the index is an aggregating index or join index. Join indexes only apply to dimension tables, so for fact tables set to `aggregating`. | +| `type` | Specifies that the index is an [aggregating index](https://docs.firebolt.io/godocs/Guides/working-with-indexes/using-aggregating-indexes.html). Should be set to `aggregating`. | | `key_column` | Sets the grouping of the aggregating index using the inputted list of column names from the model. | | `aggregation` | Sets the aggregations on the aggregating index using the inputted list of SQL agg expressions. | @@ -144,11 +144,7 @@ models: : +materialized: table +table_type: dimension - +indexes: - - type: join - join_column: - dimension_column: [ , ... ] - ... + ... ``` @@ -163,11 +159,7 @@ models: config: materialized: table table_type: dimension - indexes: - - type: join - join_column: - dimension_column: [ , ... ], - ... + ... ``` @@ -180,14 +172,7 @@ models: {{ config( materialized = "table", table_type = "dimension", - indexes = [ - { - type = "join", - join_column = "", - dimension_column: [ "", ... ] - }, - ... - ], + ... ) }} ``` @@ -195,39 +180,19 @@ models: +Dimension tables do not support aggregation indexes. #### Dimension Table Configurations | Configuration | Description | |--------------------|-------------------------------------------------------------------------------------------| | `materialized` | How the model will be materialized into Firebolt. Must be `table` to create a dimension table. | -| `table_type` | Whether the materialized table will be a [fact or dimension](https://docs.firebolt.io/working-with-tables.html#fact-and-dimension-tables) table. | -| `indexes` | A list of join indexes to create on the dimension table. | -| `type` | Specifies whether the index is an aggregating index or join index. Aggregating indexes only apply to fact tables, so for dimension tables set to `join`. | -| `join_column` | Sets the join key of the join index using the inputted column name from the model. | -| `dimension_column` | Sets the columns to be loaded into memory on the join index using the inputted list of column names from the mode. | +| `table_type` | Whether the materialized table will be a [fact or dimension](https://docs.firebolt.io/godocs/Overview/working-with-tables/working-with-tables.html#fact-and-dimension-tables) table. | -#### Example of a Dimension Table With a Join Index +## How Aggregating Indexes Are Named -``` -{{ config( - materialized = "table", - table_type = "dimension", - indexes = [ - { - type: "join", - join_column: "order_id", - dimension_column: ["customer_id", "status"] - } - ] -) }} -``` - - -## How Aggregating Indexes and Join Indexes Are Named - -In dbt-firebolt, you do not provide names for aggregating indexes and join indexes; they are named programmatically. dbt will generate index names using the following convention: +In dbt-firebolt, you do not provide names for aggregating indexes; they are named programmatically. dbt will generate index names using the following convention: ``` _____ @@ -240,7 +205,7 @@ For example, a join index could be named `my_users__id__join_1633504263` and an `dbt-firebolt` supports dbt's [external tables feature](https://docs.getdbt.com/reference/resource-properties/external), which allows dbt to manage the table ingestion process from S3 into Firebolt. This is an optional feature but can be highly convenient depending on your use case. -More information on using external tables including properly configuring IAM can be found in the Firebolt [documentation](https://docs.firebolt.io/sql-reference/commands/ddl-commands#create-external-table). +More information on using external tables including properly configuring IAM can be found in the Firebolt [documentation](https://docs.firebolt.io/godocs/Guides/loading-data/working-with-external-tables.html). #### Installation of External Tables Package @@ -288,8 +253,8 @@ sources: object_pattern: '' type: '' credentials: - internal_role_arn: arn:aws:iam::id:/ - external_role_id: + aws_key_id: + aws_secret_key: object_pattern: '' compression: '' partitions: @@ -301,6 +266,9 @@ sources: data_type: ``` +`aws_key_id` and `aws_secret_key` are the credentails that allow Firebolt access to your S3 bucket. Learn +how to set them up by following this [guide](https://docs.firebolt.io/godocs/Guides/loading-data/creating-access-keys-aws.html). If your bucket is public these parameters are not necessary. + #### Running External tables The `stage_external_sources` macro is inherited from the [dbt-external-tables package](https://github.com/dbt-labs/dbt-external-tables#syntax) and is the primary point of entry when using thes package. It has two operational modes: standard and "full refresh." @@ -311,11 +279,11 @@ $ dbt run-operation stage_external_sources # iterate through all source nodes, create or replace (no refresh command is required as data is fetched live from remote) $ dbt run-operation stage_external_sources --vars "ext_full_refresh: true" -``` +``` ## Incremental models -The [`incremental_strategy` configuration](/docs/build/incremental-strategy) controls how dbt builds incremental models. Firebolt currently supports the `append` configuration. You can specify `incremental_strategy` in `dbt_project.yml` or within a model file's `config()` block. The `append` configuration is the default. Specifying this configuration is optional. +The [`incremental_strategy` configuration](/docs/build/incremental-strategy) controls how dbt builds incremental models. Firebolt currently supports `append`, `insert_overwrite` and `delete+insert` configuration. You can specify `incremental_strategy` in `dbt_project.yml` or within a model file's `config()` block. The `append` configuration is the default. Specifying this configuration is optional. The `append` strategy performs an `INSERT INTO` statement with all the new data based on the model definition. This strategy doesn't update or delete existing rows, so if you do not filter the data to the most recent records only, it is likely that duplicate records will be inserted.