Skip to content

Commit

Permalink
docs/databricks: cost-saving using update delay and Auto Stop
Browse files Browse the repository at this point in the history
  • Loading branch information
@mdibaiee
mdibaiee committed Dec 11, 2023
1 parent 119039c commit 7a39aaa
Showing 1 changed file with 13 additions and 9 deletions.
22 changes: 13 additions & 9 deletions site/docs/reference/Connectors/materialization-connectors/databricks.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ If you haven't yet captured your data from its external source, start at the beg

You need to first create a SQL Warehouse if you don't already have one in your account. See [Databricks documentation](https://docs.databricks.com/en/sql/admin/create-sql-warehouse.html) on configuring a Databricks SQL Warehouse. After creating a SQL Warehouse, you can find the details necessary for connecting to it under the **Connection Details** tab.

In order to save on costs, we recommend that you set the Auto Stop parameter for your SQL warehouse to the minimum available. Estuary's Databricks connector automatically delays updates to the destination up to a configured Update Delay (see the endpoint configuration below), with a default value of 30 minutes. If your SQL warehouse is configured to have an Auto Stop of more than 15 minutes, we disable the automatic delay since the delay is not as effective in saving costs with a long Auto Stop idle period.

You also need an access token for your user to be used by our connector, see the respective [documentation](https://docs.databricks.com/en/administration-guide/access-control/tokens.html) from Databricks on how to create an access token.

## Configuration
Expand All @@ -38,15 +40,17 @@ Use the below properties to configure a Databricks materialization, which will d

#### Endpoint

| Property | Title | Description | Type | Required/Default |
|------------------------------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------|--------|--------------------------|
| **`/address`** | Address | Host and port of the SQL warehouse (in the form of host[:port]). Port 443 is used as the default if no specific port is provided. | string | Required |
| **`/http_path`** | HTTP Path | HTTP path of your SQL warehouse | string | Required |
| **`/catalog_name`** | Catalog Name | Name of your Unity Catalog | string | Required |
| **`/schema_name`** | Schema Name | Default schema to materialize to | string | `default` schema is used |
| **`/credentials`** | Credentials | Authentication credentials | object | |
| **`/credentials/auth_type`** | Role | Authentication type, set to `PAT` for personal access token | string | Required |
| **`/credentials/personal_access_token`** | Role | Personal Access Token | string | Required |
| Property | Title | Description | Type | Required/Default |
|------------------------------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|--------------------------|
| **`/address`** | Address | Host and port of the SQL warehouse (in the form of host[:port]). Port 443 is used as the default if no specific port is provided. | string | Required |
| **`/http_path`** | HTTP Path | HTTP path of your SQL warehouse | string | Required |
| **`/catalog_name`** | Catalog Name | Name of your Unity Catalog | string | Required |
| **`/schema_name`** | Schema Name | Default schema to materialize to | string | `default` schema is used |
| **`/credentials`** | Credentials | Authentication credentials | object | |
| **`/credentials/auth_type`** | Role | Authentication type, set to `PAT` for personal access token | string | Required |
| **`/credentials/personal_access_token`** | Role | Personal Access Token | string | Required |
| /advanced | Advanced | Options for advanced users. You should not typically need to modify these. | object | |
| /advanced/updateDelay | Update Delay | Potentially reduce active warehouse time by increasing the delay between updates. Defaults to 30 minutes if unset. | string | 30m |

#### Bindings

Expand Down

0 comments on commit 7a39aaa

Please sign in to comment.