From 538fecd72668c97d25c8ed4319be725e95987763 Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Wed, 7 Aug 2024 12:28:58 +0300 Subject: [PATCH 01/14] Add experimental supported models --- .../resource-configs/clickhouse-configs.md | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/clickhouse-configs.md b/website/docs/reference/resource-configs/clickhouse-configs.md index 32f0c81f664..0cea033a17f 100644 --- a/website/docs/reference/resource-configs/clickhouse-configs.md +++ b/website/docs/reference/resource-configs/clickhouse-configs.md @@ -13,6 +13,15 @@ id: "clickhouse-configs" | incremental materialization | YES | Creates a table if it doesn't exist, and then writes only updates to it. | | ephemeral materialized | YES | Creates a ephemeral/CTE materialization. This does model is internal to dbt and does not create any database objects | +## Experimental Models + +| Type | Supported? | Details | +|-----------------------------------------|-------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Materialized View materialization | YES, Experimental | Creates a [materialized view](https://clickhouse.com/docs/en/materialized-view). | +| Distributed table materialization | YES, Experimental | Creates a [distributed table](https://clickhouse.com/docs/en/engines/table-engines/special/distributed). Visit [this](https://clickhouse.com/docs/en/engines/table-engines/special/distributed) for more info on how it works. | +| Distributed incremental materialization | YES, Experimental | Incremental model based on the same idea as distributed table. Note that not all strategies are supported, visit [this](https://github.com/ClickHouse/dbt-clickhouse?tab=readme-ov-file#distributed-incremental-materialization) for more info. | +| Dictionary materialization | YES, Experimental | Creates a [dictionary](https://clickhouse.com/docs/en/engines/table-engines/special/dictionary). | + ### View Materialization A dbt model can be created as a [ClickHouse view](https://clickhouse.com/docs/en/sql-reference/table-functions/view/) and configured using the following syntax: @@ -222,7 +231,15 @@ For more information on configuration, check out the [snapshot configs](/referen | EmbeddedRocksDB | https://clickhouse.com/docs/en/engines/table-engines/integrations/embedded-rocksdb | | Hive | https://clickhouse.com/docs/en/engines/table-engines/integrations/hive | -If you encounter issues connecting to ClickHouse from dbt with one of the above engines, please report an issue [here](https://github.com/ClickHouse/dbt-clickhouse/issues). +## Experimental Supported Table Engines + +| Type | Details | +|-------------------|---------------------------------------------------------------------------| +| Distributed Table | https://clickhouse.com/docs/en/engines/table-engines/special/distributed. | +| Dictionary | https://clickhouse.com/docs/en/engines/table-engines/special/dictionary | + +If you encounter issues connecting to ClickHouse from dbt with one of the above engines, please report an +issue [here](https://github.com/ClickHouse/dbt-clickhouse/issues). ## Cross Database Macro Support From bde0ed66e1df91eb1ac891dfcd9ea4fbe83bbf8b Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Wed, 7 Aug 2024 12:29:49 +0300 Subject: [PATCH 02/14] listagg is now supported https://github.com/ClickHouse/dbt-clickhouse/pull/290 --- .../reference/resource-configs/clickhouse-configs.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/website/docs/reference/resource-configs/clickhouse-configs.md b/website/docs/reference/resource-configs/clickhouse-configs.md index 0cea033a17f..b96428a0974 100644 --- a/website/docs/reference/resource-configs/clickhouse-configs.md +++ b/website/docs/reference/resource-configs/clickhouse-configs.md @@ -244,9 +244,12 @@ issue [here](https://github.com/ClickHouse/dbt-clickhouse/issues). ## Cross Database Macro Support dbt-clickhouse supports most of the cross database macros now included in dbt-core, with the following exceptions: -* The `listagg` SQL function (and therefore the corresponding dbt macro) is not supported by ClickHouse. You can achieve similar results with the ClickHouse `groupArray` function but in some cases subqueries may be required to achieve the desired ordering. -* The `split_part` SQL function is implemented in ClickHouse using the splitByChar function. This function requires using a constant string for the "split" delimiter, so the `delimeter` parameter used for this macro will be interpreted as a string, not a column name -* Similarly, the `replace` SQL function in ClickHouse requires constant strings for the `old_chars` and `new_chars` parameters, so those parameters will be interpreted as strings rather than column names when invoking this macro. + +* The `split_part` SQL function is implemented in ClickHouse using the splitByChar function. This function requires + using a constant string for the "split" delimiter, so the `delimeter` parameter used for this macro will be + interpreted as a string, not a column name +* Similarly, the `replace` SQL function in ClickHouse requires constant strings for the `old_chars` and `new_chars` + parameters, so those parameters will be interpreted as strings rather than column names when invoking this macro. ## Setting `quote_columns` From 2a34cf2e973d7ee438278fb061a56d5a63418b67 Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Wed, 7 Aug 2024 12:31:06 +0300 Subject: [PATCH 03/14] add insert_overwrite as experimental supported incremental strategy --- .../resource-configs/clickhouse-configs.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/website/docs/reference/resource-configs/clickhouse-configs.md b/website/docs/reference/resource-configs/clickhouse-configs.md index b96428a0974..e3b45346218 100644 --- a/website/docs/reference/resource-configs/clickhouse-configs.md +++ b/website/docs/reference/resource-configs/clickhouse-configs.md @@ -165,16 +165,16 @@ models: #### Incremental Table Configuration -| Option | Description | Required? | -|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------| -| `materialized` | How the model will be materialized into ClickHouse. Must be `table` to create a table model. | Required | -| `unique_key` | A tuple of column names that uniquely identify rows. For more details on uniqueness constraints, see [here](/docs/build/incremental-models#defining-a-unique-key-optional). | Required. If not provided altered rows will be added twice to the incremental table. | -| `engine` | The table engine to use when creating tables. See list of supported engines below. | Optional (default: `MergeTree()`) | -| `order_by` | A tuple of column names or arbitrary expressions. This allows you to create a small sparse index that helps find data faster. | Optional (default: `tuple()`) | -| `partition_by` | A partition is a logical combination of records in a table by a specified criterion. The partition key can be any expression from the table columns. | Optional | -| `inserts_only` | (Deprecated, see the `append` materialization strategy). If True, incremental updates will be inserted directly to the target incremental table without creating an intermediate table. | Optional (default: `False`) | -| `incremental_strategy` | The strategy to use for incremental materialization. `delete+insert` and `append` are supported. For additional details on strategies, see [here](https://github.com/ClickHouse/dbt-clickhouse#incremental-model-strategies) | Optional (default: 'default') | -| `incremental_predicates` | Incremental predicate clause to be applied to `delete+insert` materializations | Optional | +| Option | Description | Required? | +|--------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------| +| `materialized` | How the model will be materialized into ClickHouse. Must be `table` to create a table model. | Required | +| `unique_key` | A tuple of column names that uniquely identify rows. For more details on uniqueness constraints, see [here](/docs/build/incremental-models#defining-a-unique-key-optional). | Required. If not provided altered rows will be added twice to the incremental table. | +| `engine` | The table engine to use when creating tables. See list of supported engines below. | Optional (default: `MergeTree()`) | +| `order_by` | A tuple of column names or arbitrary expressions. This allows you to create a small sparse index that helps find data faster. | Optional (default: `tuple()`) | +| `partition_by` | A partition is a logical combination of records in a table by a specified criterion. The partition key can be any expression from the table columns. | Optional | +| `inserts_only` | (Deprecated, see the `append` materialization strategy). If True, incremental updates will be inserted directly to the target incremental table without creating an intermediate table. | Optional (default: `False`) | +| `incremental_strategy` | The strategy to use for incremental materialization. `delete+insert`, `append` and `insert_overwrite` (experimental) are supported. For additional details on strategies, see [here](https://github.com/ClickHouse/dbt-clickhouse#incremental-model-strategies) | Optional (default: 'default') | +| `incremental_predicates` | Incremental predicate clause to be applied to `delete+insert` materializations | Optional | ## Snapshot From 6591213d0575f80097d4a70452a3a5c095970163 Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Wed, 7 Aug 2024 12:42:15 +0300 Subject: [PATCH 04/14] indentation --- .../resource-configs/clickhouse-configs.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/website/docs/reference/resource-configs/clickhouse-configs.md b/website/docs/reference/resource-configs/clickhouse-configs.md index e3b45346218..2551f871652 100644 --- a/website/docs/reference/resource-configs/clickhouse-configs.md +++ b/website/docs/reference/resource-configs/clickhouse-configs.md @@ -24,7 +24,8 @@ id: "clickhouse-configs" ### View Materialization -A dbt model can be created as a [ClickHouse view](https://clickhouse.com/docs/en/sql-reference/table-functions/view/) and configured using the following syntax: +A dbt model can be created as a [ClickHouse view](https://clickhouse.com/docs/en/sql-reference/table-functions/view/) +and configured using the following syntax: @@ -253,7 +259,8 @@ dbt-clickhouse supports most of the cross database macros now included in dbt-co ## Setting `quote_columns` -To prevent a warning, make sure to explicitly set a value for `quote_columns` in your `dbt_project.yml`. See the [doc on quote_columns](https://docs.getdbt.com/reference/resource-configs/quote_columns) for more information. +To prevent a warning, make sure to explicitly set a value for `quote_columns` in your `dbt_project.yml`. See +the [doc on quote_columns](https://docs.getdbt.com/reference/resource-configs/quote_columns) for more information. ```yaml seeds: From fd4f56e1b11748004d9da7a1c764f724acd00653 Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Wed, 7 Aug 2024 12:46:11 +0300 Subject: [PATCH 05/14] extend clickhouse-dbt authors --- .../docs/docs/core/connect-data-platform/clickhouse-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/core/connect-data-platform/clickhouse-setup.md b/website/docs/docs/core/connect-data-platform/clickhouse-setup.md index fce367be812..89cf11070a1 100644 --- a/website/docs/docs/core/connect-data-platform/clickhouse-setup.md +++ b/website/docs/docs/core/connect-data-platform/clickhouse-setup.md @@ -3,7 +3,7 @@ title: "ClickHouse setup" description: "Read this guide to learn about the ClickHouse warehouse setup in dbt." meta: maintained_by: Community - authors: 'Geoff Genz' + authors: 'Geoff Genz & Bentsi Leviav' github_repo: 'ClickHouse/dbt-clickhouse' pypi_package: 'dbt-clickhouse' min_core_version: 'v0.19.0' From 2ff77fcf10337fcdd5f9d3f24b0fc72304315189 Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Wed, 7 Aug 2024 12:46:35 +0300 Subject: [PATCH 06/14] update profile example to include latest additions --- .../connect-data-platform/clickhouse-setup.md | 99 +++++++++++-------- 1 file changed, 56 insertions(+), 43 deletions(-) diff --git a/website/docs/docs/core/connect-data-platform/clickhouse-setup.md b/website/docs/docs/core/connect-data-platform/clickhouse-setup.md index 89cf11070a1..61902c38223 100644 --- a/website/docs/docs/core/connect-data-platform/clickhouse-setup.md +++ b/website/docs/docs/core/connect-data-platform/clickhouse-setup.md @@ -33,55 +33,68 @@ To connect to ClickHouse from dbt, you'll need to add a [profile](https://docs.g outputs: : type: clickhouse - schema: - user: - password: - #optional fields - driver: http|native - port: - host: - retries: 1 - verify: False - secure: True - connect_timeout: 10 - send_receive_timeout: 300 - sync_request_timeout: 5 - compression: False - compress_block_size: 1048576 - database_engine: - check_exchange: True - use_lw_deletes: False - custom_settings: - + schema: [ default ] # ClickHouse database for dbt models + + # optional + driver: [ http ] # http or native. If not set this will be auto-determined based on port setting + host: [ localhost ] + port: [ 8123 ] # If not set, defaults to 8123, 8443, 9000, 9440 depending on the secure and driver settings + user: [ default ] # User for all database operations + password: [ ] # Password for the user + cluster: [ ] # If set, certain DDL/table operations will be executed with the `ON CLUSTER` clause using this cluster. Distributed materializations require this setting to work. See the following ClickHouse Cluster section for more details. + verify: [ True ] # Validate TLS certificate if using TLS/SSL + secure: [ False ] # Use TLS (native protocol) or HTTPS (http protocol) + retries: [ 1 ] # Number of times to retry a "retriable" database exception (such as a 503 'Service Unavailable' error) + compression: [ ] # Use gzip compression if truthy (http), or compression type for a native connection + connect_timeout: [ 10 ] # Timeout in seconds to establish a connection to ClickHouse + send_receive_timeout: [ 300 ] # Timeout in seconds to receive data from the ClickHouse server + cluster_mode: [ False ] # Use specific settings designed to improve operation on Replicated databases (recommended for ClickHouse Cloud) + use_lw_deletes: [ False ] # Use the strategy `delete+insert` as the default incremental strategy. + check_exchange: [ True ] # Validate that clickhouse support the atomic EXCHANGE TABLES command. (Not needed for most ClickHouse versions) + local_suffix: [ _local ] # Table suffix of local tables on shards for distributed materializations. + local_db_prefix: [ ] # Database prefix of local tables on shards for distributed materializations. If empty, it uses the same database as the distributed table. + allow_automatic_deduplication: [ False ] # Enable ClickHouse automatic deduplication for Replicated tables + tcp_keepalive: [ False ] # Native client only, specify TCP keepalive configuration. Specify custom keepalive settings as [idle_time_sec, interval_sec, probes]. + custom_settings: [ { } ] # A dictionary/mapping of custom ClickHouse settings for the connection - default is empty. + + # Native (clickhouse-driver) connection settings + sync_request_timeout: [ 5 ] # Timeout for server ping + compress_block_size: [ 1048576 ] # Compression block size if compression is enabled + + ``` #### Description of ClickHouse Profile Fields - -| Field | Description | -|------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `type` | This must be included either in `profiles.yml` or in the `dbt_project.yml` file. Must be set to `clickhouse`. | -| `schema` | Required. A ClickHouse's database name. The dbt model database.schema.table is not compatible with ClickHouse because ClickHouse does not support a schema. So we use a simple model schema.table, where schema is the ClickHouse's database. We don't recommend using the `default` database. | -| `user` | Required. A ClickHouse username with adequate permissions to access the specified `schema`. | -| `password` | Required. The password associated with the specified `user`. | -| `driver` | Optional. The ClickHouse client interface, `http` or `native`. Defaults to `http` unless the `port` is set to 9440 or 9400, in which case the `native` driver is assumed. | -| `port` | Optional. ClickHouse server port number. Defaults to 8123/8443 (secure) if the driver is `http`, and to 9000/9440(secure) if the driver is `native`. | -| `host` | Optional. The host name of the connection. Default is `localhost`. | -| `retries` | Optional. Number of times to retry the initial connection attempt if the error appears to be recoverable. | -| `verify` | Optional. For (`secure=True`) connections, validate the ClickHouse server TLS certificate, including matching hostname, expiration, and signed by a trusted Certificate Authority. Defaults to True. | -| `secure` | Optional. Whether the connection (either http or native) is secured by TLS. This converts an http driver connection to https, and a native driver connection to the native ClickHouse protocol over TLS. the Defaults to False. | -| `cluster_mode` | Optional. Add connection settings to improve compatibility with clusters using the Replicated Database Engine. Default False. | -| `connect_timeout` | Optional. Connection timeout in seconds. Defaults is 10 seconds. | -| `send_receive_timeout` | Optional. Timeout for receiving data from or sending data to ClickHouse. Defaults to 5 minutes (300 seconds) | -| `sync_request_timeout` | Optional. Timeout for connection ping request (native connection only). Defaults to 5 seconds. | -| `compression` | Optional. Use compression in the connection. Defaults to `False`. If set to `True` for HTTP, this enables gzip compression. If set to `True` for the native protocol, this enabled lz4 compression. Other valid values are `lz4hc` and `zstd` for the native driver only. | -| `compress_block_size` | Optional. Compression block size (in bytes) when using compression with the native driver. Defaults to 1MB | -| `database_engine` | Optional. Database engine to use when creating new ClickHouse schemas (databases). If not set (the default), new databases will use the default ClickHouse database engine (usually Atomic). | -| `check_exchange` | Optional. On connecting to the ClickHouse, if this is parameter is `True` DBT will validate that the ClickHouse server supports atomic exchange of tables. Using atomic exchange (when available) improves reliability and parallelism. This check is unnecessary for ClickHouse running on recent Linux operating system, and in those circumstances can be disabled by setting `check_exchange` to `False` to avoid additional overhead on startup. Defaults to `True`. | -| `use_lw_deletes` | Optional. If ClickHouse experimental lightweight deletes are available, use the `delete+insert` strategy as the default strategy for incremental materializations. Defaults to `False` (use legacy strategy). | -| `custom_settings` | Optional. A mapping of ClickHouse specific user settings to use with the connection. See the ClickHouse documentation for supported settings. | +| Field | Description | +|---------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `type` | This must be included either in `profiles.yml` or in the `dbt_project.yml` file. Must be set to `clickhouse`. | +| `schema` | Required. A ClickHouse's database name. The dbt model database.schema.table is not compatible with ClickHouse because ClickHouse does not support a schema. So we use a simple model schema.table, where schema is the ClickHouse's database. We don't recommend using the `default` database. | +| `driver` | Optional. The ClickHouse client interface, `http` or `native`. Defaults to `http` unless the `port` is set to 9440 or 9400, in which case the `native` driver is assumed. | +| `host` | Optional. The host name of the connection. Default is `localhost`. | +| `port` | Optional. ClickHouse server port number. Defaults to 8123/8443 (secure) if the driver is `http`, and to 9000/9440(secure) if the driver is `native`. | +| `user` | Required. A ClickHouse username with adequate permissions to access the specified `schema`. | +| `password` | Required. The password associated with the specified `user`. | +| `cluster` | Optional. If set, certain DDL/table operations will be executed with the `ON CLUSTER` clause using this cluster. Distributed materializations require this setting to work. See the following ClickHouse Cluster section for more details. | +| `verify` | Optional. For (`secure=True`) connections, validate the ClickHouse server TLS certificate, including matching hostname, expiration, and signed by a trusted Certificate Authority. Defaults to True. | +| `secure` | Optional. Whether the connection (either http or native) is secured by TLS. This converts an http driver connection to https, and a native driver connection to the native ClickHouse protocol over TLS. the Defaults to False. | +| `retries` | Optional. Number of times to retry the initial connection attempt if the error appears to be recoverable. | +| `compression` | Optional. Use compression in the connection. Defaults to `False`. If set to `True` for HTTP, this enables gzip compression. If set to `True` for the native protocol, this enabled lz4 compression. Other valid values are `lz4hc` and `zstd` for the native driver only. | +| `connect_timeout` | Optional. Connection timeout in seconds. Defaults is 10 seconds. | +| `send_receive_timeout` | Optional. Timeout for receiving data from or sending data to ClickHouse. Defaults to 5 minutes (300 seconds) | +| `cluster_mode` | Optional. Add connection settings to improve compatibility with clusters using the Replicated Database Engine. Default False. | +| `use_lw_deletes` | Optional. If ClickHouse experimental lightweight deletes are available, use the `delete+insert` strategy as the default strategy for incremental materializations. Defaults to `False` (use legacy strategy). | +| `check_exchange` | Optional. On connecting to the ClickHouse, if this is parameter is `True` DBT will validate that the ClickHouse server supports atomic exchange of tables. Using atomic exchange (when available) improves reliability and parallelism. This check is unnecessary for ClickHouse running on recent Linux operating system, and in those circumstances can be disabled by setting `check_exchange` to `False` to avoid additional overhead on startup. Defaults to `True`. | +| `local_suffix` | Optional. Table suffix of local tables on shards for distributed materializations. Defaults to '_local'. | +| `local_db_prefix` | Optional. Database prefix of local tables on shards for distributed materializations. If empty, it uses the same database as the distributed table. Defaults to empty string. | +| `allow_automatic_deduplication` | Optional. Enable ClickHouse automatic deduplication for Replicated tables. Defaults to False. | +| `tcp_keepalive` | Optional. Native client only, specify TCP keepalive configuration. Specify custom keepalive settings as `idle_time_sec`, `interval_sec`, `probes`. Defaults to False. | +| `sync_request_timeout` | Optional. Timeout for connection ping request (native connection only). Defaults to 5 seconds. | +| `compress_block_size` | Optional. Compression block size (in bytes) when using compression with the native driver. Defaults to 1MB | +| `database_engine` | Optional. Database engine to use when creating new ClickHouse schemas (databases). If not set (the default), new databases will use the default ClickHouse database engine (usually Atomic). | +| `custom_settings` | Optional. A mapping of ClickHouse specific user settings to use with the connection. See the ClickHouse documentation for supported settings. | #### Troubleshooting Connections From 51c1ee8cc250080819a39d202cbc1f9290833341 Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Wed, 7 Aug 2024 12:49:22 +0300 Subject: [PATCH 07/14] indentation --- .../core/connect-data-platform/clickhouse-setup.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/website/docs/docs/core/connect-data-platform/clickhouse-setup.md b/website/docs/docs/core/connect-data-platform/clickhouse-setup.md index 61902c38223..6a5e0d53b66 100644 --- a/website/docs/docs/core/connect-data-platform/clickhouse-setup.md +++ b/website/docs/docs/core/connect-data-platform/clickhouse-setup.md @@ -15,7 +15,8 @@ meta: config_page: '/reference/resource-configs/clickhouse-configs' --- -Some core functionality may be limited. If you're interested in contributing, check out the source code for each repository listed below. +Some core functionality may be limited. If you're interested in contributing, check out the source code for each +repository listed below. import SetUpPages from '/snippets/_setup-pages-intro.md'; @@ -23,7 +24,8 @@ import SetUpPages from '/snippets/_setup-pages-intro.md'; ## Connecting to ClickHouse with **dbt-clickhouse** -To connect to ClickHouse from dbt, you'll need to add a [profile](https://docs.getdbt.com/docs/core/connection-profiles) to your `profiles.yml` file. A ClickHouse profile conforms to the following syntax: +To connect to ClickHouse from dbt, you'll need to add a [profile](https://docs.getdbt.com/docs/core/connection-profiles) +to your `profiles.yml` file. A ClickHouse profile conforms to the following syntax: @@ -99,6 +101,9 @@ To connect to ClickHouse from dbt, you'll need to add a [profile](https://docs.g #### Troubleshooting Connections If you encounter issues connecting to ClickHouse from dbt, make sure the following criteria are met: -- The engine must be one of the [supported engines](/reference/resource-configs/clickhouse-configs#supported-table-engines). + +- The engine must be one of + the [supported engines](/reference/resource-configs/clickhouse-configs#supported-table-engines). - You must have adequate permissions to access the database. -- If you're not using the default table engine for the database, you must specify a table engine in your model configuration. +- If you're not using the default table engine for the database, you must specify a table engine in your model + configuration. From 3ce236127cf3a219ed16d8bd80607927541423da Mon Sep 17 00:00:00 2001 From: bentsileviav Date: Wed, 7 Aug 2024 13:01:21 +0300 Subject: [PATCH 08/14] remove double link --- .../resource-configs/clickhouse-configs.md | 38 +++++++++---------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/website/docs/reference/resource-configs/clickhouse-configs.md b/website/docs/reference/resource-configs/clickhouse-configs.md index 2551f871652..95a7e045669 100644 --- a/website/docs/reference/resource-configs/clickhouse-configs.md +++ b/website/docs/reference/resource-configs/clickhouse-configs.md @@ -18,7 +18,7 @@ id: "clickhouse-configs" | Type | Supported? | Details | |-----------------------------------------|-------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Materialized View materialization | YES, Experimental | Creates a [materialized view](https://clickhouse.com/docs/en/materialized-view). | -| Distributed table materialization | YES, Experimental | Creates a [distributed table](https://clickhouse.com/docs/en/engines/table-engines/special/distributed). Visit [this](https://clickhouse.com/docs/en/engines/table-engines/special/distributed) for more info on how it works. | +| Distributed table materialization | YES, Experimental | Creates a [distributed table](https://clickhouse.com/docs/en/engines/table-engines/special/distributed). | | Distributed incremental materialization | YES, Experimental | Incremental model based on the same idea as distributed table. Note that not all strategies are supported, visit [this](https://github.com/ClickHouse/dbt-clickhouse?tab=readme-ov-file#distributed-incremental-materialization) for more info. | | Dictionary materialization | YES, Experimental | Creates a [dictionary](https://clickhouse.com/docs/en/engines/table-engines/special/dictionary). | @@ -28,12 +28,12 @@ A dbt model can be created as a [ClickHouse view](https://clickhouse.com/docs/en and configured using the following syntax: @@ -65,12 +65,12 @@ A dbt model can be created as a [ClickHouse table](https://clickhouse.com/docs/e configured using the following syntax: @@ -122,12 +122,12 @@ sets or complex transformations. To address this challenge and reduce the build incremental ClickHouse table and is configured using the following syntax: From ddbae70f719daa23b89230767c80e729be94d462 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 7 Aug 2024 12:23:16 -0400 Subject: [PATCH 09/14] Apply suggestions from code review Editorial changes --- .../docs/core/connect-data-platform/clickhouse-setup.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/docs/core/connect-data-platform/clickhouse-setup.md b/website/docs/docs/core/connect-data-platform/clickhouse-setup.md index 6a5e0d53b66..d2b641b2387 100644 --- a/website/docs/docs/core/connect-data-platform/clickhouse-setup.md +++ b/website/docs/docs/core/connect-data-platform/clickhouse-setup.md @@ -38,12 +38,12 @@ to your `profiles.yml` file. A ClickHouse profile conforms to the following synt schema: [ default ] # ClickHouse database for dbt models # optional - driver: [ http ] # http or native. If not set this will be auto-determined based on port setting + driver: [ http ] # http or native. If not configured, this will be auto-determined based on the port setting host: [ localhost ] - port: [ 8123 ] # If not set, defaults to 8123, 8443, 9000, 9440 depending on the secure and driver settings + port: [ 8123 ] # Defaults to 8123, 8443, 9000, 9440 depending on the secure and driver settings user: [ default ] # User for all database operations password: [ ] # Password for the user - cluster: [ ] # If set, certain DDL/table operations will be executed with the `ON CLUSTER` clause using this cluster. Distributed materializations require this setting to work. See the following ClickHouse Cluster section for more details. + cluster: [ ] # If configured, certain DDL/table operations will be executed with the `ON CLUSTER` clause using this cluster. Distributed materializations require this setting to work. See the following ClickHouse Cluster section for more details. verify: [ True ] # Validate TLS certificate if using TLS/SSL secure: [ False ] # Use TLS (native protocol) or HTTPS (http protocol) retries: [ 1 ] # Number of times to retry a "retriable" database exception (such as a 503 'Service Unavailable' error) From 077c0e91083da62ed8b6c6690b601f7eb83df450 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 7 Aug 2024 14:00:00 -0400 Subject: [PATCH 10/14] Update clickhouse-setup.md Fixing spacing and formatting --- .../docs/core/connect-data-platform/clickhouse-setup.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/website/docs/docs/core/connect-data-platform/clickhouse-setup.md b/website/docs/docs/core/connect-data-platform/clickhouse-setup.md index d2b641b2387..f1114f1505b 100644 --- a/website/docs/docs/core/connect-data-platform/clickhouse-setup.md +++ b/website/docs/docs/core/connect-data-platform/clickhouse-setup.md @@ -68,7 +68,7 @@ to your `profiles.yml` file. A ClickHouse profile conforms to the following synt -#### Description of ClickHouse Profile Fields +### Description of ClickHouse Profile Fields | Field | Description | |---------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| @@ -98,12 +98,11 @@ to your `profiles.yml` file. A ClickHouse profile conforms to the following synt | `database_engine` | Optional. Database engine to use when creating new ClickHouse schemas (databases). If not set (the default), new databases will use the default ClickHouse database engine (usually Atomic). | | `custom_settings` | Optional. A mapping of ClickHouse specific user settings to use with the connection. See the ClickHouse documentation for supported settings. | -#### Troubleshooting Connections +## Troubleshooting Connections If you encounter issues connecting to ClickHouse from dbt, make sure the following criteria are met: -- The engine must be one of - the [supported engines](/reference/resource-configs/clickhouse-configs#supported-table-engines). +- The engine must be one of the [supported engines](/reference/resource-configs/clickhouse-configs#supported-table-engines). - You must have adequate permissions to access the database. - If you're not using the default table engine for the database, you must specify a table engine in your model configuration. From 5e582943e0afcfd0c77f41cf1c39d6837cc7fb02 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 7 Aug 2024 14:07:12 -0400 Subject: [PATCH 11/14] Apply suggestions from code review --- website/docs/reference/resource-configs/clickhouse-configs.md | 1 + 1 file changed, 1 insertion(+) diff --git a/website/docs/reference/resource-configs/clickhouse-configs.md b/website/docs/reference/resource-configs/clickhouse-configs.md index 95a7e045669..7ce0c81c003 100644 --- a/website/docs/reference/resource-configs/clickhouse-configs.md +++ b/website/docs/reference/resource-configs/clickhouse-configs.md @@ -14,6 +14,7 @@ id: "clickhouse-configs" | ephemeral materialized | YES | Creates a ephemeral/CTE materialization. This does model is internal to dbt and does not create any database objects | ## Experimental Models +The following are [experimental features](https://clickhouse.com/docs/en/beta-and-experimental-features) in Clickhouse: | Type | Supported? | Details | |-----------------------------------------|-------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| From 1a2534365c3f043a7b270a165b7bedc24afe1124 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 7 Aug 2024 14:15:46 -0400 Subject: [PATCH 12/14] Update website/docs/reference/resource-configs/clickhouse-configs.md --- website/docs/reference/resource-configs/clickhouse-configs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/clickhouse-configs.md b/website/docs/reference/resource-configs/clickhouse-configs.md index 7ce0c81c003..dbf7b34d7a8 100644 --- a/website/docs/reference/resource-configs/clickhouse-configs.md +++ b/website/docs/reference/resource-configs/clickhouse-configs.md @@ -238,7 +238,7 @@ For more information on configuration, check out the [snapshot configs](/referen | EmbeddedRocksDB | https://clickhouse.com/docs/en/engines/table-engines/integrations/embedded-rocksdb | | Hive | https://clickhouse.com/docs/en/engines/table-engines/integrations/hive | -## Experimental Supported Table Engines +## Experimental supported table engines | Type | Details | |-------------------|---------------------------------------------------------------------------| From a3a9444d01d536d0d253e1dbbe99e92e40517977 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 7 Aug 2024 14:17:11 -0400 Subject: [PATCH 13/14] Update clickhouse-configs.md Changing titles to sentence case per style guide --- .../resource-configs/clickhouse-configs.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/website/docs/reference/resource-configs/clickhouse-configs.md b/website/docs/reference/resource-configs/clickhouse-configs.md index dbf7b34d7a8..ef011402eb4 100644 --- a/website/docs/reference/resource-configs/clickhouse-configs.md +++ b/website/docs/reference/resource-configs/clickhouse-configs.md @@ -13,7 +13,7 @@ id: "clickhouse-configs" | incremental materialization | YES | Creates a table if it doesn't exist, and then writes only updates to it. | | ephemeral materialized | YES | Creates a ephemeral/CTE materialization. This does model is internal to dbt and does not create any database objects | -## Experimental Models +## Experimental models The following are [experimental features](https://clickhouse.com/docs/en/beta-and-experimental-features) in Clickhouse: | Type | Supported? | Details | @@ -23,7 +23,7 @@ The following are [experimental features](https://clickhouse.com/docs/en/beta-an | Distributed incremental materialization | YES, Experimental | Incremental model based on the same idea as distributed table. Note that not all strategies are supported, visit [this](https://github.com/ClickHouse/dbt-clickhouse?tab=readme-ov-file#distributed-incremental-materialization) for more info. | | Dictionary materialization | YES, Experimental | Creates a [dictionary](https://clickhouse.com/docs/en/engines/table-engines/special/dictionary). | -### View Materialization +### View materialization A dbt model can be created as a [ClickHouse view](https://clickhouse.com/docs/en/sql-reference/table-functions/view/) and configured using the following syntax: @@ -60,7 +60,7 @@ models: -### Table Materialization +### Table materialization A dbt model can be created as a [ClickHouse table](https://clickhouse.com/docs/en/operations/system-tables/tables/) and configured using the following syntax: @@ -107,7 +107,7 @@ models: -#### Table Configuration +#### Table configuration | Option | Description | Required? | |----------------|------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------| @@ -116,7 +116,7 @@ models: | `order_by` | A tuple of column names or arbitrary expressions. This allows you to create a small sparse index that helps find data faster. | Optional (default: `tuple()`) | | `partition_by` | A partition is a logical combination of records in a table by a specified criterion. The partition key can be any expression from the table columns. | Optional | -### Incremental Materialization +### Incremental materialization Table model will be reconstructed for each dbt execution. This may be infeasible and extremely costly for larger result sets or complex transformations. To address this challenge and reduce the build time, a dbt model can be created as an @@ -168,7 +168,7 @@ models: -#### Incremental Table Configuration +#### Incremental table configuration | Option | Description | Required? | |--------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------| @@ -227,7 +227,7 @@ supported by the ClickHouse connector and is configured using the following synt For more information on configuration, check out the [snapshot configs](/reference/snapshot-configs) reference page. -## Supported Table Engines +## Supported table engines | Type | Details | |------------------------|-------------------------------------------------------------------------------------------| @@ -248,7 +248,7 @@ For more information on configuration, check out the [snapshot configs](/referen If you encounter issues connecting to ClickHouse from dbt with one of the above engines, please report an issue [here](https://github.com/ClickHouse/dbt-clickhouse/issues). -## Cross Database Macro Support +## Cross database macro support dbt-clickhouse supports most of the cross database macros now included in dbt-core, with the following exceptions: From aae47a904a66557d251b6b9f906addb927bbad30 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 7 Aug 2024 14:18:30 -0400 Subject: [PATCH 14/14] Update clickhouse-configs.md Spacing fix --- website/docs/reference/resource-configs/clickhouse-configs.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/website/docs/reference/resource-configs/clickhouse-configs.md b/website/docs/reference/resource-configs/clickhouse-configs.md index ef011402eb4..02096bb8f23 100644 --- a/website/docs/reference/resource-configs/clickhouse-configs.md +++ b/website/docs/reference/resource-configs/clickhouse-configs.md @@ -260,8 +260,7 @@ dbt-clickhouse supports most of the cross database macros now included in dbt-co ## Setting `quote_columns` -To prevent a warning, make sure to explicitly set a value for `quote_columns` in your `dbt_project.yml`. See -the [doc on quote_columns](https://docs.getdbt.com/reference/resource-configs/quote_columns) for more information. +To prevent a warning, make sure to explicitly set a value for `quote_columns` in your `dbt_project.yml`. See the [doc on quote_columns](https://docs.getdbt.com/reference/resource-configs/quote_columns) for more information. ```yaml seeds: