diff --git a/website/docs/docs/build/semantic-models.md b/website/docs/docs/build/semantic-models.md index 2dc550d5a4a..ffb5e9653f7 100644 --- a/website/docs/docs/build/semantic-models.md +++ b/website/docs/docs/build/semantic-models.md @@ -109,6 +109,29 @@ semantic_models: type: categorical ``` + + +Semantic models support configs in either the schema file or at the project level. + +Semantic model config in `models/semantic.yml`: +```yml +semantic_models: + - name: orders + config: + enabled: true | false + group: some_group +``` + +Semantic model config in `dbt_project.yml`: +```yml +semantic_models: + my_project_name: + +enabled: true | false + +group: some_group +``` + + + ### Name Define the name of the semantic model. You must define a unique name for the semantic model. The semantic graph will use this name to identify the model, and you can update it at any time. Avoid using double underscores (__) in the name as they're not supported. diff --git a/website/docs/docs/cloud/dbt-cloud-ide/lint-format.md b/website/docs/docs/cloud/dbt-cloud-ide/lint-format.md index 8ffd83ef00e..6a86f1aa14b 100644 --- a/website/docs/docs/cloud/dbt-cloud-ide/lint-format.md +++ b/website/docs/docs/cloud/dbt-cloud-ide/lint-format.md @@ -45,7 +45,11 @@ With the dbt Cloud IDE, you can seamlessly use [SQLFluff](https://sqlfluff.com/) - Works with Jinja and SQL, - Comes with built-in [linting rules](https://docs.sqlfluff.com/en/stable/rules.html). You can also [customize](#customize-linting) your own linting rules. - Empowers you to [enable linting](#enable-linting) with options like **Lint** (displays linting errors and recommends actions) or **Fix** (auto-fixes errors in the IDE). -- Displays a **Code Quality** tab to view code errors, and provides code quality visibility and management. +- Displays a **Code Quality** tab to view code errors, and provides code quality visibility and management. + +:::info Ephemeral models not supported +Linting doesn't support ephemeral models in dbt v1.5 and lower. Refer to the [FAQs](#faqs) for more info. +::: ### Enable linting @@ -223,6 +227,12 @@ Currently, running SQLFluff commands from the terminal isn't supported. Make sure you're on a development branch. Formatting or Linting isn't available on "main" or "read-only" branches. +
+Why is there inconsistent SQLFluff behavior when running outside the dbt Cloud IDE (such as a GitHub Action)? +— Double-check your SQLFluff version matches the one in dbt Cloud IDE (found in the Code Quality tab after a lint operation).

+— If your lint operation passes despite clear rule violations, confirm you're not linting models with ephemeral models. Linting doesn't support ephemeral models in dbt v1.5 and lower. +
+ ## Related docs - [User interface](/docs/cloud/dbt-cloud-ide/ide-user-interface) diff --git a/website/docs/docs/collaborate/govern/project-dependencies.md b/website/docs/docs/collaborate/govern/project-dependencies.md index 1dbc967e74e..7785e428678 100644 --- a/website/docs/docs/collaborate/govern/project-dependencies.md +++ b/website/docs/docs/collaborate/govern/project-dependencies.md @@ -100,3 +100,11 @@ There are a few cases where installing another internal project as a package can - Coordinated changes — In development, if you wanted to test the effects of a change to a public model in an upstream project (`jaffle_finance.monthly_revenue`) on a downstream model (`jaffle_marketing.roi_by_channel`) _before_ introducing changes to a staging or production environment, you can install the `jaffle_finance` package as a package within `jaffle_marketing`. The installation can point to a specific git branch, however, if you find yourself frequently needing to perform end-to-end testing across both projects, we recommend you re-examine if this represents a stable interface boundary. These are the exceptions, rather than the rule. Installing another team's project as a package adds complexity, latency, and risk of unnecessary costs. By defining clear interface boundaries across teams, by serving one team's public models as "APIs" to another, and by enabling practitioners to develop with a more narrowly-defined scope, we can enable more people to contribute, with more confidence, while requiring less context upfront. + +## FAQs + +
+Can I define private packages in the dependencies.yml file? + +If you're using private packages with the [git token method](/docs/build/packages#git-token-method), you must define them in the `packages.yml` file instead of the `dependencies.yml` file. This is because conditional rendering (like Jinja-in-yaml) is not supported. +
diff --git a/website/docs/docs/core/connect-data-platform/trino-setup.md b/website/docs/docs/core/connect-data-platform/trino-setup.md index 70ecdbdfc96..39d8ed8ab3f 100644 --- a/website/docs/docs/core/connect-data-platform/trino-setup.md +++ b/website/docs/docs/core/connect-data-platform/trino-setup.md @@ -83,7 +83,7 @@ The following profile fields are optional to set up. They let you configure your | Profile field | Example | Description | | ----------------------------- | -------------------------------- | ----------------------------------------------------------------------------------------------------------- | | `threads` | `8` | How many threads dbt should use (default is `1`) | -| `roles` | `system: analyst` | Catalog roles can be set under the optional `roles` parameter using following format: `catalog: role`. | +| `roles` | `system: analyst` | Catalog roles can be set under the optional `roles` parameter using the following format: `catalog: role`. | | `session_properties` | `query_max_run_time: 4h` | Sets Trino session properties used in the connection. Execute `SHOW SESSION` to see available options | | `prepared_statements_enabled` | `true` or `false` | Enable usage of Trino prepared statements (used in `dbt seed` commands) (default: `true`) | | `retries` | `10` | Configure how many times all database operation is retried when connection issues arise (default: `3`) | diff --git a/website/docs/docs/dbt-cloud-apis/user-tokens.md b/website/docs/docs/dbt-cloud-apis/user-tokens.md index e56d8b2f974..f0d694f5edd 100644 --- a/website/docs/docs/dbt-cloud-apis/user-tokens.md +++ b/website/docs/docs/dbt-cloud-apis/user-tokens.md @@ -13,7 +13,7 @@ permissions of the user the that they were created for. You can find your User API token in the Profile page under the `API Access` label. - + ## FAQs diff --git a/website/docs/docs/use-dbt-semantic-layer/gsheets.md b/website/docs/docs/use-dbt-semantic-layer/gsheets.md index 8ddaae0364c..c4480f63923 100644 --- a/website/docs/docs/use-dbt-semantic-layer/gsheets.md +++ b/website/docs/docs/use-dbt-semantic-layer/gsheets.md @@ -51,3 +51,10 @@ To use the filter functionality, choose the dimension you want to filter by and - If it's a categorical dimension, type in the dimension value you want to filter by (no quotes needed) and press enter. - Continue adding additional filters as needed with AND and OR. If it's a time dimension, choose the operator and select from the calendar. + + +**Limited Use Policy Disclosure** + +The dbt Semantic Layer for Sheet's use and transfer to any other app of information received from Google APIs will adhere to [Google API Services User Data Policy](https://developers.google.com/terms/api-services-user-data-policy), including the Limited Use requirements. + + diff --git a/website/docs/faqs/Models/reference-models-in-another-project.md b/website/docs/faqs/Models/reference-models-in-another-project.md deleted file mode 100644 index 19f3f52da31..00000000000 --- a/website/docs/faqs/Models/reference-models-in-another-project.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -title: How can I reference models or macros in another project? -description: "Use packages to add another project to your dbt project" -sidebar_label: 'Reference models or macros in another project' -id: reference-models-in-another-project - ---- - -You can use [packages](/docs/build/packages) to add another project to your dbt -project, including other projects you've created. Check out the [docs](/docs/build/packages) -for more information! diff --git a/website/docs/reference/commands/cmd-docs.md b/website/docs/reference/commands/cmd-docs.md index 754c5e93baf..bc4840464b8 100644 --- a/website/docs/reference/commands/cmd-docs.md +++ b/website/docs/reference/commands/cmd-docs.md @@ -19,6 +19,18 @@ The command is responsible for generating your project's documentation website b dbt docs generate ``` + + +Use the `--select` argument to limit the nodes included within `catalog.json`. When this flag is provided, step (3) will be restricted to the selected nodes. All other nodes will be excluded. Step (2) is unaffected. + +**Example**: +```shell +dbt docs generate --select +orders +``` + + + + Use the `--no-compile` argument to skip re-compilation. When this flag is provided, `dbt docs generate` will skip step (2) described above. **Example**: diff --git a/website/docs/reference/commands/init.md b/website/docs/reference/commands/init.md index 873647814ec..ac55717c0ec 100644 --- a/website/docs/reference/commands/init.md +++ b/website/docs/reference/commands/init.md @@ -17,10 +17,21 @@ Then, it will: - Create a new folder with your project name and sample files, enough to get you started with dbt - Create a connection profile on your local machine. The default location is `~/.dbt/profiles.yml`. Read more in [configuring your profile](/docs/core/connect-data-platform/connection-profiles). + + +When using `dbt init` to initialize your project, include the `--profile` flag to specify an existing `profiles.yml` as the `profile:` key to use instead of creating a new one. For example, `dbt init --profile`. + + + +If the profile does not exist in `profiles.yml` or the command is run inside an existing project, the command raises an error. + + + ## Existing project If you've just cloned or downloaded an existing dbt project, `dbt init` can still help you set up your connection profile so that you can start working quickly. It will prompt you for connection information, as above, and add a profile (using the `profile` name from the project) to your local `profiles.yml`, or create the file if it doesn't already exist. + ## profile_template.yml `dbt init` knows how to prompt for connection information by looking for a file named `profile_template.yml`. It will look for this file in two places: diff --git a/website/docs/reference/node-selection/test-selection-examples.md b/website/docs/reference/node-selection/test-selection-examples.md index 52439d95d97..478aac932c5 100644 --- a/website/docs/reference/node-selection/test-selection-examples.md +++ b/website/docs/reference/node-selection/test-selection-examples.md @@ -19,14 +19,14 @@ Run generic tests only: ```bash - $ dbt test --select test_type:generic + dbt test --select test_type:generic ``` Run singular tests only: ```bash - $ dbt test --select test_type:singular + dbt test --select test_type:singular ``` In both cases, `test_type` checks a property of the test itself. These are forms of "direct" test selection. @@ -87,8 +87,8 @@ By default, a test will run when ANY parent is selected; we call this "eager" in In this mode, any test that depends on unbuilt resources will raise an error. ```shell -$ dbt test --select orders -$ dbt build --select orders + dbt test --select orders + dbt build --select orders ``` @@ -102,8 +102,8 @@ It will only include tests whose references are each within the selected nodes. Put another way, it will prevent tests from running if one or more of its parents is unselected. ```shell -$ dbt test --select orders --indirect-selection=cautious -$ dbt build --select orders --indirect-selection=cautious + dbt test --select orders --indirect-selection=cautious + dbt build --select orders --indirect-selection=cautious ``` @@ -122,8 +122,8 @@ By default, a test will run when ANY parent is selected; we call this "eager" in In this mode, any test that depends on unbuilt resources will raise an error. ```shell -$ dbt test --select orders -$ dbt build --select orders + dbt test --select orders + dbt build --select orders ``` @@ -137,8 +137,8 @@ It will only include tests whose references are each within the selected nodes. Put another way, it will prevent tests from running if one or more of its parents is unselected. ```shell -$ dbt test --select orders --indirect-selection=cautious -$ dbt build --select orders --indirect-selection=cautious + dbt test --select orders --indirect-selection=cautious + dbt build --select orders --indirect-selection=cautious ``` @@ -152,8 +152,8 @@ It will only include tests whose references are each within the selected nodes ( This is useful in the same scenarios as "cautious", but also includes when a test depends on a model **and** a direct ancestor of that model (like confirming an aggregation has the same totals as its input). ```shell -$ dbt test --select orders --indirect-selection=buildable -$ dbt build --select orders --indirect-selection=buildable + dbt test --select orders --indirect-selection=buildable + dbt build --select orders --indirect-selection=buildable ``` @@ -172,8 +172,8 @@ By default, a test will run when ANY parent is selected; we call this "eager" in In this mode, any test that depends on unbuilt resources will raise an error. ```shell -$ dbt test --select orders -$ dbt build --select orders + dbt test --select orders + dbt build --select orders ``` @@ -187,8 +187,8 @@ It will only include tests whose references are each within the selected nodes. Put another way, it will prevent tests from running if one or more of its parents is unselected. ```shell -$ dbt test --select orders --indirect-selection=cautious -$ dbt build --select orders --indirect-selection=cautious + dbt test --select orders --indirect-selection=cautious + dbt build --select orders --indirect-selection=cautious ``` @@ -202,8 +202,8 @@ It will only include tests whose references are each within the selected nodes ( This is useful in the same scenarios as "cautious", but also includes when a test depends on a model **and** a direct ancestor of that model (like confirming an aggregation has the same totals as its input). ```shell -$ dbt test --select orders --indirect-selection=buildable -$ dbt build --select orders --indirect-selection=buildable +dbt test --select orders --indirect-selection=buildable +dbt build --select orders --indirect-selection=buildable ``` @@ -213,8 +213,8 @@ $ dbt build --select orders --indirect-selection=buildable This mode will only include tests whose references are each within the selected nodes and will ignore all tests from attached nodes. ```shell -$ dbt test --select orders --indirect-selection=empty -$ dbt build --select orders --indirect-selection=empty + dbt test --select orders --indirect-selection=empty + dbt build --select orders --indirect-selection=empty ``` @@ -234,22 +234,25 @@ The following examples should feel somewhat familiar if you're used to executing ```bash # Run tests on a model (indirect selection) - $ dbt test --select customers + dbt test --select customers + + # Run tests on two or more specific models (indirect selection) + dbt test --select customers orders # Run tests on all models in the models/staging/jaffle_shop directory (indirect selection) - $ dbt test --select staging.jaffle_shop + dbt test --select staging.jaffle_shop # Run tests downstream of a model (note this will select those tests directly!) - $ dbt test --select stg_customers+ + dbt test --select stg_customers+ # Run tests upstream of a model (indirect selection) - $ dbt test --select +stg_customers + dbt test --select +stg_customers # Run tests on all models with a particular tag (direct + indirect) - $ dbt test --select tag:my_model_tag + dbt test --select tag:my_model_tag # Run tests on all models with a particular materialization (indirect selection) - $ dbt test --select config.materialized:table + dbt test --select config.materialized:table ``` @@ -258,16 +261,19 @@ The following examples should feel somewhat familiar if you're used to executing ```bash # tests on all sources - $ dbt test --select source:* + dbt test --select source:* # tests on one source - $ dbt test --select source:jaffle_shop + dbt test --select source:jaffle_shop + + # tests on two or more specific sources + dbt test --select source:jaffle_shop source:raffle_bakery # tests on one source table - $ dbt test --select source:jaffle_shop.customers + dbt test --select source:jaffle_shop.customers # tests on everything _except_ sources - $ dbt test --exclude source:* + dbt test --exclude source:* ``` ### More complex selection @@ -276,10 +282,10 @@ Through the combination of direct and indirect selection, there are many ways to ```bash - $ dbt test --select assert_total_payment_amount_is_positive # directly select the test by name - $ dbt test --select payments,test_type:singular # indirect selection, v1.2 - $ dbt test --select payments,test_type:data # indirect selection, v0.18.0 - $ dbt test --select payments --data # indirect selection, earlier versions + dbt test --select assert_total_payment_amount_is_positive # directly select the test by name + dbt test --select payments,test_type:singular # indirect selection, v1.2 + dbt test --select payments,test_type:data # indirect selection, v0.18.0 + dbt test --select payments --data # indirect selection, earlier versions ``` @@ -288,13 +294,13 @@ Through the combination of direct and indirect selection, there are many ways to ```bash # Run tests on all models with a particular materialization - $ dbt test --select config.materialized:table + dbt test --select config.materialized:table # Run tests on all seeds, which use the 'seed' materialization - $ dbt test --select config.materialized:seed + dbt test --select config.materialized:seed # Run tests on all snapshots, which use the 'snapshot' materialization - $ dbt test --select config.materialized:snapshot + dbt test --select config.materialized:snapshot ``` Note that this functionality may change in future versions of dbt. @@ -322,7 +328,7 @@ models: ```bash - $ dbt test --select tag:my_column_tag + dbt test --select tag:my_column_tag ``` Currently, tests "inherit" tags applied to columns, sources, and source tables. They do _not_ inherit tags applied to models, seeds, or snapshots. In all likelihood, those tests would still be selected indirectly, because the tag selects its parent. This is a subtle distinction, and it may change in future versions of dbt. @@ -350,5 +356,5 @@ models: ```bash - $ dbt test --select tag:my_test_tag + dbt test --select tag:my_test_tag ``` diff --git a/website/snippets/quickstarts/schedule-a-job.md b/website/snippets/quickstarts/schedule-a-job.md index 59d428bdfaa..ab8f4350dbf 100644 --- a/website/snippets/quickstarts/schedule-a-job.md +++ b/website/snippets/quickstarts/schedule-a-job.md @@ -24,15 +24,16 @@ Jobs are a set of dbt commands that you want to run on a schedule. For example, As the `jaffle_shop` business gains more customers, and those customers create more orders, you will see more records added to your source data. Because you materialized the `customers` model as a table, you'll need to periodically rebuild your table to ensure that the data stays up-to-date. This update will happen when you run a job. -1. After creating your deployment environment, you should be directed to the page for new environment. If not, select **Deploy** in the upper left, then click **Jobs**. -2. Click **Create one** and provide a name, for example "Production run", and link to the Environment you just created. -3. Scroll down to "Execution Settings" and select **Generate docs on run**. -4. Under "Commands," add this command as part of your job if you don't see them: - * `dbt build` -5. For this exercise, do _not_ set a schedule for your project to run — while your organization's project should run regularly, there's no need to run this example project on a schedule. Scheduling a job is sometimes referred to as _deploying a project_. -6. Select **Save**, then click **Run now** to run your job. -7. Click the run and watch its progress under "Run history." -8. Once the run is complete, click **View Documentation** to see the docs for your project. +1. After creating your deployment environment, you should be directed to the page for a new environment. If not, select **Deploy** in the upper left, then click **Jobs**. +2. Click **Create one** and provide a name, for example, "Production run", and link to the Environment you just created. +3. Scroll down to the **Execution Settings** section. +4. Under **Commands**, add this command as part of your job if you don't see it: + * `dbt build` +5. Select the **Generate docs on run** checkbox to automatically [generate updated project docs](/docs/collaborate/build-and-view-your-docs) each time your job runs. +6. For this exercise, do _not_ set a schedule for your project to run — while your organization's project should run regularly, there's no need to run this example project on a schedule. Scheduling a job is sometimes referred to as _deploying a project_. +7. Select **Save**, then click **Run now** to run your job. +8. Click the run and watch its progress under "Run history." +9. Once the run is complete, click **View Documentation** to see the docs for your project. :::tip Congratulations 🎉! You've just deployed your first dbt project! diff --git a/website/static/img/Filtering.png b/website/static/img/Filtering.png index 5a15a59f23e..b05394bd459 100644 Binary files a/website/static/img/Filtering.png and b/website/static/img/Filtering.png differ diff --git a/website/static/img/Paginate.png b/website/static/img/Paginate.png index 84a15732c12..21e2fd138b8 100644 Binary files a/website/static/img/Paginate.png and b/website/static/img/Paginate.png differ diff --git a/website/static/img/api-access-profile.jpg b/website/static/img/api-access-profile.jpg new file mode 100644 index 00000000000..36ffd4beda8 Binary files /dev/null and b/website/static/img/api-access-profile.jpg differ diff --git a/website/static/img/api-access-profile.png b/website/static/img/api-access-profile.png deleted file mode 100644 index deade9f2135..00000000000 Binary files a/website/static/img/api-access-profile.png and /dev/null differ diff --git a/website/static/img/dbt-cloud-project-setup-flow-next.png b/website/static/img/dbt-cloud-project-setup-flow-next.png index 660e8ae446a..92f46bccd0a 100644 Binary files a/website/static/img/dbt-cloud-project-setup-flow-next.png and b/website/static/img/dbt-cloud-project-setup-flow-next.png differ diff --git a/website/static/img/delete_projects_from_dbt_cloud_20221023.gif b/website/static/img/delete_projects_from_dbt_cloud_20221023.gif index 246c912c55b..b579556d457 100644 Binary files a/website/static/img/delete_projects_from_dbt_cloud_20221023.gif and b/website/static/img/delete_projects_from_dbt_cloud_20221023.gif differ diff --git a/website/static/img/node_color_example.png b/website/static/img/node_color_example.png index 83b26f5735a..a1a62742ca0 100644 Binary files a/website/static/img/node_color_example.png and b/website/static/img/node_color_example.png differ diff --git a/website/static/img/sample_email_data.png b/website/static/img/sample_email_data.png deleted file mode 100644 index 7224d42e60b..00000000000 Binary files a/website/static/img/sample_email_data.png and /dev/null differ diff --git a/website/vercel.json b/website/vercel.json index c5fb0638fba..072062ec939 100644 --- a/website/vercel.json +++ b/website/vercel.json @@ -2,6 +2,16 @@ "cleanUrls": true, "trailingSlash": false, "redirects": [ + { + "source": "/faqs/models/reference-models-in-another-project", + "destination": "/docs/collaborate/govern/project-dependencies", + "permanent": true + }, + { + "source": "/faqs/Models/reference-models-in-another-project", + "destination": "/docs/collaborate/govern/project-dependencies", + "permanent": true + }, { "source": "/docs/deploy/job-triggers", "destination": "/docs/deploy/deploy-jobs",