Skip to content

Commit

Permalink
Update Firebolt features and connection parameters
Browse files Browse the repository at this point in the history
  • Loading branch information
@ptiurin
ptiurin committed Jul 31, 2024
1 parent 13fa550 commit e0b05d8
Show file tree
Hide file tree
Showing 2 changed files with 29 additions and 65 deletions.
26 changes: 11 additions & 15 deletions website/docs/docs/core/connect-data-platform/firebolt-setup.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,47 +40,43 @@ To connect to Firebolt from dbt, you'll need to add a [profile](https://docs.get
outputs:
<target-name>:
type: firebolt
user: "<username>"
password: "<password>"
client_id: "<id>"
client_secret: "<secret>"
database: "<database-name>"
engine_name: "<engine-name>"
account_name: "<account-name>"
schema: <tablename-prefix>
threads: 1
#optional fields
jar_path: <path-to-local-jdbc-driver>
host: "<hostname>"
account_name: "<account-name>"
```
</File>
#### Description of Firebolt Profile Fields
To specify values as environment variables, use the format `{{ env_var('<variable_name>' }}`. For example, `{{ env_var('DATABASE_NAME' }}`.
To specify values as environment variables, use the format `{{ env_var('<variable_name>' }}`. For example, `{{ env_var('DATABASE_NAME' }}`.

| Field | Description |
|--------------------------|--------------------------------------------------------------------------------------------------------|
| `type` | This must be included either in `profiles.yml` or in the `dbt_project.yml` file. Must be set to `firebolt`. |
| `user` | Required. A Firebolt username with adequate permissions to access the specified `engine_name`. |
| `password` | Required. The password associated with the specified `user`. |
| `client_id` | Required. Your [service account](https://docs.firebolt.io/godocs/Guides/managing-your-organization/service-accounts.html) id. |
| `client_secret` | Required. The secret associated with the specified `client_id`. |
| `database` | Required. The name of the Firebolt database to connect to. |
| `engine_name` | Required in version 0.21.10 and later. Optional in earlier versions. The name (not the URL) of the Firebolt engine to use in the specified `database`. This must be a general purpose read-write engine and the engine must be running. If omitted in earlier versions, the default engine for the specified `database` is used. |
| `account_name` | Required. Specifies the account name under which the specified `database` exists. |
| `schema` | Recommended. A string to add as a prefix to the names of generated tables when using the [custom schemas workaround](https://docs.getdbt.com/reference/warehouse-profiles/firebolt-profile#supporting-concurrent-development). |
| `threads` | Required. Must be set to `1`. Multi-threading is not currently supported. |
| `jar_path` | Required only with versions earlier than 0.21.0. Ignored in 0.21.0 and later. The path to your JDBC driver on your local drive. |
| `threads` | Required. Set to higher number to improve performance. |
| `host` | Optional. The host name of the connection. For all customers it is `api.app.firebolt.io`, which will be used if omitted. |
| `account_name` | Required if more than one account is associated with the specified `user1`. Specifies the account name (not the account ID) under which the specified `database` exists. If omitted, the default account is assumed. |


#### Troubleshooting Connections

If you encounter issues connecting to Firebolt from dbt, make sure the following criteria are met:
- The engine must be a general-purpose read-write engine, not an analytics engine.
- You must have adequate permissions to access the engine.
- You must have adequate permissions to access the engine and the database.
- Your service account must be attached to a user.
- The engine must be running.
- If you're not using the default engine for the database, you must specify an engine name.
- If there is more than one account associated with your credentials, you must specify an account.


## Supporting Concurrent Development
Expand Down
68 changes: 18 additions & 50 deletions website/docs/reference/resource-configs/firebolt-configs.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ models:
table_type: fact
primary_index: [ <column-name>, ... ]
indexes:
- type: aggregating | join
- type: aggregating
key_column: [ <column-name>, ... ]
aggregation: [ <agg-sql>, ... ]
...
Expand Down Expand Up @@ -96,10 +96,10 @@ models:
| Configuration | Description |
|-------------------|-------------------------------------------------------------------------------------------|
| `materialized` | How the model will be materialized into Firebolt. Must be `table` to create a fact table. |
| `table_type` | Whether the materialized table will be a [fact or dimension](https://docs.firebolt.io/working-with-tables.html#fact-and-dimension-tables) table. |
| `table_type` | Whether the materialized table will be a [fact or dimension](https://docs.firebolt.io/godocs/Overview/working-with-tables/working-with-tables.html#fact-and-dimension-tables) table. |
| `primary_index` | Sets the primary index for the fact table using the inputted list of column names from the model. Required for fact tables. |
| `indexes` | A list of aggregating indexes to create on the fact table. |
| `type` | Specifies whether the index is an aggregating index or join index. Join indexes only apply to dimension tables, so for fact tables set to `aggregating`. |
| `type` | Specifies that the index is an [aggregating index](https://docs.firebolt.io/godocs/Guides/working-with-indexes/using-aggregating-indexes.html). Should be set to `aggregating`. |
| `key_column` | Sets the grouping of the aggregating index using the inputted list of column names from the model. |
| `aggregation` | Sets the aggregations on the aggregating index using the inputted list of SQL agg expressions. |

Expand Down Expand Up @@ -144,11 +144,7 @@ models:
<resource-path>:
+materialized: table
+table_type: dimension
+indexes:
- type: join
join_column: <column-name>
dimension_column: [ <column-name>, ... ]
...
...
```

</File>
Expand All @@ -163,11 +159,7 @@ models:
config:
materialized: table
table_type: dimension
indexes:
- type: join
join_column: <column-name>
dimension_column: [ <column-name>, ... ],
...
...
```

</File>
Expand All @@ -180,54 +172,27 @@ models:
{{ config(
materialized = "table",
table_type = "dimension",
indexes = [
{
type = "join",
join_column = "<column-name>",
dimension_column: [ "<column-name>", ... ]
},
...
],
...
) }}
```

</File>
</TabItem>
</Tabs>

Dimension tables do not support aggregation indexes.

#### Dimension Table Configurations

| Configuration | Description |
|--------------------|-------------------------------------------------------------------------------------------|
| `materialized` | How the model will be materialized into Firebolt. Must be `table` to create a dimension table. |
| `table_type` | Whether the materialized table will be a [fact or dimension](https://docs.firebolt.io/working-with-tables.html#fact-and-dimension-tables) table. |
| `indexes` | A list of join indexes to create on the dimension table. |
| `type` | Specifies whether the index is an aggregating index or join index. Aggregating indexes only apply to fact tables, so for dimension tables set to `join`. |
| `join_column` | Sets the join key of the join index using the inputted column name from the model. |
| `dimension_column` | Sets the columns to be loaded into memory on the join index using the inputted list of column names from the mode. |
| `table_type` | Whether the materialized table will be a [fact or dimension](https://docs.firebolt.io/godocs/Overview/working-with-tables/working-with-tables.html#fact-and-dimension-tables) table. |


#### Example of a Dimension Table With a Join Index
## How Aggregating Indexes Are Named

```
{{ config(
materialized = "table",
table_type = "dimension",
indexes = [
{
type: "join",
join_column: "order_id",
dimension_column: ["customer_id", "status"]
}
]
) }}
```


## How Aggregating Indexes and Join Indexes Are Named

In dbt-firebolt, you do not provide names for aggregating indexes and join indexes; they are named programmatically. dbt will generate index names using the following convention:
In dbt-firebolt, you do not provide names for aggregating indexes; they are named programmatically. dbt will generate index names using the following convention:

```
<table-name>__<key-column>__<index-type>_<unix-timestamp-at-execution>
Expand All @@ -240,7 +205,7 @@ For example, a join index could be named `my_users__id__join_1633504263` and an

`dbt-firebolt` supports dbt's [external tables feature](https://docs.getdbt.com/reference/resource-properties/external), which allows dbt to manage the table ingestion process from S3 into Firebolt. This is an optional feature but can be highly convenient depending on your use case.

More information on using external tables including properly configuring IAM can be found in the Firebolt [documentation](https://docs.firebolt.io/sql-reference/commands/ddl-commands#create-external-table).
More information on using external tables including properly configuring IAM can be found in the Firebolt [documentation](https://docs.firebolt.io/godocs/Guides/loading-data/working-with-external-tables.html).


#### Installation of External Tables Package
Expand Down Expand Up @@ -288,8 +253,8 @@ sources:
object_pattern: '<regex>'
type: '<type>'
credentials:
internal_role_arn: arn:aws:iam::id:<role>/<bucket-name>
external_role_id: <external-id>
aws_key_id: <key-id>
aws_secret_key: <key-secret>
object_pattern: '<regex>'
compression: '<compression-type>'
partitions:
Expand All @@ -301,6 +266,9 @@ sources:
data_type: <type>
```

`aws_key_id` and `aws_secret_key` are the credentails that allow Firebolt access to your S3 bucket. Learn
how to set them up by following this [guide](https://docs.firebolt.io/godocs/Guides/loading-data/creating-access-keys-aws.html). If your bucket is public these parameters are not necessary.

#### Running External tables

The `stage_external_sources` macro is inherited from the [dbt-external-tables package](https://github.com/dbt-labs/dbt-external-tables#syntax) and is the primary point of entry when using thes package. It has two operational modes: standard and "full refresh."
Expand All @@ -311,11 +279,11 @@ $ dbt run-operation stage_external_sources
# iterate through all source nodes, create or replace (no refresh command is required as data is fetched live from remote)
$ dbt run-operation stage_external_sources --vars "ext_full_refresh: true"
```
```

## Incremental models

The [`incremental_strategy` configuration](/docs/build/incremental-strategy) controls how dbt builds incremental models. Firebolt currently supports the `append` configuration. You can specify `incremental_strategy` in `dbt_project.yml` or within a model file's `config()` block. The `append` configuration is the default. Specifying this configuration is optional.
The [`incremental_strategy` configuration](/docs/build/incremental-strategy) controls how dbt builds incremental models. Firebolt currently supports `append`, `insert_overwrite` and `delete+insert` configuration. You can specify `incremental_strategy` in `dbt_project.yml` or within a model file's `config()` block. The `append` configuration is the default. Specifying this configuration is optional.

The `append` strategy performs an `INSERT INTO` statement with all the new data based on the model definition. This strategy doesn't update or delete existing rows, so if you do not filter the data to the most recent records only, it is likely that duplicate records will be inserted.

Expand Down

0 comments on commit e0b05d8

Please sign in to comment.