-
Notifications
You must be signed in to change notification settings - Fork 982
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
c915480
commit 227a979
Showing
4 changed files
with
227 additions
and
6 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
103 changes: 103 additions & 0 deletions
103
website/docs/docs/core/connect-data-platform/starrocks-setup.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| --- | ||
| title: "Starrocks setup" | ||
| description: "Read this guide to learn about the Starrocks warehouse setup in dbt." | ||
| id: "starrocks-setup" | ||
| meta: | ||
| maintained_by: Starrocks | ||
| authors: Astralidea | ||
| github_repo: 'StarRocks/starrocks/tree/main/contrib/dbt-connector' | ||
| pypi_package: 'dbt-starrocks' | ||
| min_core_version: 'v1.6.2' | ||
| min_supported_version: 'Starrocks 2.5' | ||
| cloud_support: Not Supported | ||
| slack_channel_name: '#db-starrocks' | ||
| slack_channel_link: 'https://www.getdbt.com/community' | ||
| platform_name: 'Starrocks' | ||
| config_page: '/reference/resource-configs/starrocks-configs' | ||
| --- | ||
|
|
||
| <h2> Overview of {frontMatter.meta.pypi_package} </h2> | ||
|
|
||
| <ul> | ||
| <li><strong>Maintained by</strong>: {frontMatter.meta.maintained_by}</li> | ||
| <li><strong>Authors</strong>: {frontMatter.meta.authors}</li> | ||
| <li><strong>GitHub repo</strong>: <a href={`https://github.com/${frontMatter.meta.github_repo}`}>{frontMatter.meta.github_repo}</a><a href={`https://github.com/${frontMatter.meta.github_repo}`}><img src={`https://img.shields.io/github/stars/${frontMatter.meta.github_repo}?style=for-the-badge`}/></a></li> | ||
| <li><strong>PyPI package</strong>: <code>{frontMatter.meta.pypi_package}</code> <a href={`https://badge.fury.io/py/${frontMatter.meta.pypi_package}`}><img src={`https://badge.fury.io/py/${frontMatter.meta.pypi_package}.svg`}/></a></li> | ||
| <li><strong>Slack channel</strong>: <a href={frontMatter.meta.slack_channel_link}>{frontMatter.meta.slack_channel_name}</a></li> | ||
| <li><strong>Supported dbt Core version</strong>: {frontMatter.meta.min_core_version} and newer</li> | ||
| <li><strong>dbt Cloud support</strong>: {frontMatter.meta.cloud_support}</li> | ||
| <li><strong>Minimum data platform version</strong>: {frontMatter.meta.min_supported_version}</li> | ||
| </ul> | ||
|
|
||
|
|
||
| <h2> Installing {frontMatter.meta.pypi_package} </h2> | ||
|
|
||
| pip is the easiest way to install the adapter: | ||
|
|
||
| <code>pip install {frontMatter.meta.pypi_package}</code> | ||
|
|
||
| <p>Installing <code>{frontMatter.meta.pypi_package}</code> will also install <code>dbt-core</code> and any other dependencies.</p> | ||
|
|
||
| <h2> Configuring {frontMatter.meta.pypi_package} </h2> | ||
|
|
||
| <p>For {frontMatter.meta.platform_name}-specifc configuration please refer to <a href={frontMatter.meta.config_page}>{frontMatter.meta.platform_name} Configuration</a> </p> | ||
|
|
||
| <p>For further info, refer to the GitHub repository: <a href={`https://github.com/${frontMatter.meta.github_repo}`}>{frontMatter.meta.github_repo}</a></p> | ||
|
|
||
|
|
||
| ## Authentication Methods | ||
|
|
||
| ### User / Password Authentication | ||
|
|
||
| Starrocks can be configured using basic user/password authentication as shown below. | ||
|
|
||
| <File name='~/.dbt/profiles.yml'> | ||
|
|
||
| ```yaml | ||
| my-starrocks-db: | ||
| target: dev | ||
| outputs: | ||
| dev: | ||
| type: starrocks | ||
| host: localhost | ||
| port: 9030 | ||
| schema: analytics | ||
|
|
||
| # User/password auth | ||
| username: your_starrocks_username | ||
| password: your_starrocks_password | ||
| ``` | ||
| </File> | ||
| #### Description of Profile Fields | ||
| | Option | Description | Required? | Example | | ||
| |----------|--------------------------------------------------------|-----------|--------------------------------| | ||
| | type | The specific adapter to use | Required | `starrocks` | | ||
| | host | The hostname to connect to | Required | `192.168.100.28` | | ||
| | port | The port to use | Required | `9030` | | ||
| | schema | Specify the schema (database) to build models into | Required | `analytics` | | ||
| | username | The username to use to connect to the server | Required | `dbt_admin` | | ||
| | password | The password to use for authenticating to the server | Required | `correct-horse-battery-staple` | | ||
| | version | Let Plugin try to go to a compatible starrocks version | Optional | `3.1.0` | | ||
|
|
||
| ## Supported features | ||
|
|
||
| | Starrocks <= 2.5 | Starrocks 2.5 ~ 3.1 | Starrocks >= 3.1 | Feature | | ||
| |:----------------:|:--------------------:|:-----------------:|:---------------------------------:| | ||
| | ✅ | ✅ | ✅ | Table materialization | | ||
| | ✅ | ✅ | ✅ | View materialization | | ||
| | ❌ | ❌ | ✅ | Materialized View materialization | | ||
| | ❌ | ✅ | ✅ | Incremental materialization | | ||
| | ❌ | ✅ | ✅ | Primary Key Model | | ||
| | ✅ | ✅ | ✅ | Sources | | ||
| | ✅ | ✅ | ✅ | Custom data tests | | ||
| | ✅ | ✅ | ✅ | Docs generate | | ||
| | ❌ | ❌ | ❌ | Kafka | | ||
|
|
||
| ### Notice | ||
| 1. When StarRocks Version < 2.5, `Create table as` can only set engine='OLAP' and table_type='DUPLICATE' | ||
| 2. When StarRocks Version >= 2.5, `Create table as` supports table_type='PRIMARY' | ||
| 3. When StarRocks Version < 3.1 distributed_by is required | ||
|
|
||
| It is recommended to use the latest starrocks version and dbt-starrocks version for the best experience. |
116 changes: 116 additions & 0 deletions
116
website/docs/reference/resource-configs/starrocks-configs.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,116 @@ | ||
| --- | ||
| title: "Starrocks configurations" | ||
| id: "starrocks-configs" | ||
| description: "Starrocks Configurations - Read this in-depth guide to learn about configurations in dbt." | ||
| --- | ||
|
|
||
| ## Model Configuration | ||
|
|
||
| A dbt model can be configured using the following syntax: | ||
|
|
||
| <Tabs | ||
| groupId="config-fact" | ||
| defaultValue="project-yaml" | ||
| values={[ | ||
| { label: 'Project file', value: 'project-yaml', }, | ||
| { label: 'Property file', value: 'property-yaml', }, | ||
| { label: 'Config block', value: 'config', }, | ||
| ] | ||
| }> | ||
|
|
||
|
|
||
| <TabItem value="project-yaml"> | ||
| <File name='dbt_project.yml'> | ||
|
|
||
| ```yaml | ||
| models: | ||
| <resource-path>: | ||
| materialized: table // table or view or materialized_view | ||
| keys: ['id', 'name', 'some_date'] | ||
| table_type: 'PRIMARY' // PRIMARY or DUPLICATE or UNIQUE | ||
| distributed_by: ['id'] | ||
| buckets: 3 // default 10 | ||
| partition_by: ['some_date'] | ||
| partition_by_init: ["PARTITION p1 VALUES [('1971-01-01 00:00:00'), ('1991-01-01 00:00:00')),PARTITION p1972 VALUES [('1991-01-01 00:00:00'), ('1999-01-01 00:00:00'))"] | ||
| properties: [{"replication_num":"1", "in_memory": "true"}] | ||
| refresh_method: 'async' // only for materialized view default manual | ||
| ``` | ||
| </File> | ||
| </TabItem> | ||
| <TabItem value="property-yaml"> | ||
| <File name='models/properties.yml'> | ||
| ```yaml | ||
| models: | ||
| - name: <model-name> | ||
| config: | ||
| materialized: table // table or view or materialized_view | ||
| keys: ['id', 'name', 'some_date'] | ||
| table_type: 'PRIMARY' // PRIMARY or DUPLICATE or UNIQUE | ||
| distributed_by: ['id'] | ||
| buckets: 3 // default 10 | ||
| partition_by: ['some_date'] | ||
| partition_by_init: ["PARTITION p1 VALUES [('1971-01-01 00:00:00'), ('1991-01-01 00:00:00')),PARTITION p1972 VALUES [('1991-01-01 00:00:00'), ('1999-01-01 00:00:00'))"] | ||
| properties: [{"replication_num":"1", "in_memory": "true"}] | ||
| refresh_method: 'async' // only for materialized view default manual | ||
| ``` | ||
| </File> | ||
| </TabItem> | ||
| <TabItem value="config"> | ||
| <File name='models/<model_name>.sql'> | ||
| ```jinja | ||
| {{ config( | ||
| materialized = 'table', | ||
| keys=['id', 'name', 'some_date'], | ||
| table_type='PRIMARY', | ||
| distributed_by=['id'], | ||
| buckets=3, | ||
| partition_by=['some_date'], | ||
| .... | ||
| ) }} | ||
| ``` | ||
| </File> | ||
| </TabItem> | ||
| </Tabs> | ||
|
|
||
| ### Configuration Description | ||
|
|
||
| | Option | Description | | ||
| |---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | `materialized` | How the model will be materialized into Starrocks. Supports view, table, incremental, ephemeral, and materialized_view. | | ||
| | `keys` | Which columns serve as keys. | | ||
| | `table_type` | Table type, supported are PRIMARY or DUPLICATE or UNIQUE. | | ||
| | `distributed_by` | Specifies the column of data distribution. If not specified, it defaults to random. | | ||
| | `buckets` | The bucket number in one partition. If not specified, it will be automatically inferred. | | ||
| | `partition_by` | The partition column list. | | ||
| | `partition_by_init` | The partition rule or some real partitions item. | | ||
| | `properties` | The table properties configuration of Starrocks. ([Starrocks table properties](https://docs.starrocks.io/en-us/latest/sql-reference/sql-statements/data-definition/CREATE_TABLE#properties)) | | ||
| | `refresh_method` | How to refresh materialized views. | | ||
|
|
||
| ## Read From Catalog | ||
| First you need to add this catalog to starrocks. The following is an example of hive. | ||
|
|
||
| ```sql | ||
| CREATE EXTERNAL CATALOG `hive_catalog` | ||
| PROPERTIES ( | ||
| "hive.metastore.uris" = "thrift://127.0.0.1:8087", | ||
| "type"="hive" | ||
| ); | ||
| ``` | ||
| How to add other types of catalogs can be found in the documentation. [Catalog Overview](https://docs.starrocks.io/en-us/latest/data_source/catalog/catalog_overview) Then write the sources.yaml file. | ||
| ```yaml | ||
| sources: | ||
| - name: external_example | ||
| schema: hive_catalog.hive_db | ||
| tables: | ||
| - name: hive_table_name | ||
| ``` | ||
| Finally, you might use below marco quote | ||
| ```jinja | ||
| {{ source('external_example', 'hive_table_name') }} | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters