Skip to content

Commit

Permalink
Merge branch 'hook-change' of https://github.com/dbt-labs/docs.getdbt…
Browse files Browse the repository at this point in the history
….com into hook-change
  • Loading branch information
matthewshaver committed Oct 2, 2024
2 parents b493b5c + ebe1bb1 commit 1b34dbe
Show file tree
Hide file tree
Showing 8 changed files with 70 additions and 7 deletions.
4 changes: 3 additions & 1 deletion website/docs/docs/collaborate/govern/project-dependencies.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,9 @@ For more guidance on how to use dbt Mesh, refer to the dedicated [dbt Mesh guide

### Safeguarding production data with staging environments

When working in a Development environment, cross-project `ref`s normally resolve to the Production environment of the project. However, to protect production data, set up a [Staging deployment environment](/docs/deploy/deploy-environments#staging-environment) within your projects. With a staging environment integrated into the project, any references from external projects during development workflows resolve to the Staging environment. This adds a layer of security between your Deployment and Production environments by limiting access to production data.
When working in a Development environment, cross-project `ref`s normally resolve to the Production environment of the project. However, to protect production data, set up a [Staging deployment environment](/docs/deploy/deploy-environments#staging-environment) within your projects.

With a staging environment integrated into the project, dbt Mesh automatically fetches public model information from the producer’s staging environment if the consumer is also in staging. Similarly, dbt Mesh fetches from the producer’s production environment if the consumer is in production. This ensures consistency between environments and adds a layer of security by preventing access to production data during development workflows.

Read [Why use a staging environment](/docs/deploy/deploy-environments#why-use-a-staging-environment) for more information about the benefits.

Expand Down
4 changes: 2 additions & 2 deletions website/docs/docs/dbt-support.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ Types of dbt Cloud-related questions our Support team can assist you with, regar

Basic assistance with dbt project troubleshooting.
Help with errors and issues in macros, models, and dbt Labs' packages.
For strategic advice, expansion, and project setup, consult Solutions Architect and Sales Director.
For strategic advice, best practices, or expansion conversations, consult your Account Team.

For customers on a dbt Cloud Enterprise plan, we **also** offer basic assistance in troubleshooting issues with your dbt project:
- **Something isn't working the way I would expect it to...**
Expand All @@ -76,7 +76,7 @@ For customers on a dbt Cloud Enterprise plan, we **also** offer basic assistance
- `Compilation Error Error reading name_of_folder/name_of_file.yml - Runtime Error Syntax
error near line 9`

Types of questions you should ask your Solutions Architect and Sales Director:
Types of questions you should ask your Account Team:
- How should we think about setting up our dbt projects, environments, and jobs based on our company structure and needs?
- I want to expand my account! How do I add more people and train them?
- Here is our data road map for the next year - can we talk through how dbt fits into it and what features we may not be utilizing that can help us achieve our goals?
Expand Down
2 changes: 2 additions & 0 deletions website/docs/docs/dbt-versions/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ Release notes are grouped by month for both multi-tenant and virtual private clo
- **Behavior change:** Set [`state_modified_compare_more_unrendered`](reference/global-configs/behavior-changes#source-definitions-for-state) to true to reduce false positives for `state:modified` when configs differ between `dev` and `prod` environments.
- **Behavior change:** Set the [`skip_nodes_if_on_run_start_fails`](/reference/global-configs/behavior-changes#on-run-start-hook) flag to `True` to skip all selected resources from running if there is a failure on an `on-run-start` hook.
- **Enhancement**: In dbt Cloud Versionless, snapshots defined in SQL files can now use `config` defined in `schema.yml` YAML files. This update resolves the previous limitation that required snapshot properties to be defined exclusively in `dbt_project.yml` and/or a `config()` block within the SQL file. This enhancement will be included in the upcoming dbt Core v1.9 release.
- **Enhancement**: In May 2024, dbt Cloud versionless began inferring a model's `primary_key` based on configured data tests and/or constraints within `manifest.json`. The inferred `primary_key` is visible in dbt Explorer and utilized by the dbt Cloud [compare changes](/docs/deploy/run-visibility#compare-tab) feature. This will also be released in dbt Core 1.9.
Read about the [order dbt infers columns can be used as primary key of a model](https://github.com/dbt-labs/dbt-core/blob/7940ad5c7858ff11ef100260a372f2f06a86e71f/core/dbt/contracts/graph/nodes.py#L534-L541).
- **New:** dbt Explorer now includes trust signal icons, which is currently available as a [Preview](/docs/dbt-versions/product-lifecycles#dbt-cloud). Trust signals offer a quick, at-a-glance view of data health when browsing your dbt models in Explorer. These icons indicate whether a model is **Healthy**, **Caution**, **Degraded**, or **Unknown**. For accurate health data, ensure the resource is up-to-date and has had a recent job run. Refer to [Trust signals](/docs/collaborate/explore-projects#trust-signals-for-resources) for more information.
- **New:** Auto exposures are now available in Preview in dbt Cloud. Auto-exposures helps users understand how their models are used in downstream analytics tools to inform investments and reduce incidents. It imports and auto-generates exposures based on Tableau dashboards, with user-defined curation. To learn more, refer to [Auto exposures](/docs/collaborate/auto-exposures).

Expand Down
23 changes: 23 additions & 0 deletions website/docs/reference/global-configs/adapter-behavior-changes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
title: "About adapter-specific behavior changes"
id: "adapter-behavior-changes"
sidebar_label: "Adapter behavior changes"
description: "Adapter-specific behavior changes"
hide_table_of_contents: true
pagination_next: null
pagination_prev: null
---


Some adapters can display behavior changes when certain flags are enabled. The following sections contain details about these adapter-specific behavior changes.


<div className="grid--3-col">

<Card
title="Redshift"
body="Behavior changes for the Amazon Redshift adapter."
link="reference/global-configs/redshift-changes"
icon="redshift"/>

</div>
11 changes: 11 additions & 0 deletions website/docs/reference/global-configs/redshift-changes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
title: "Amazon Redshift adapter behavior changes"
id: "redshift-changes"
sidebar: "Redshift"
---

## The restrict_direct_pg_catalog_access flag

Originally, the `dbt-redshift` adapter was built on top of the `dbt-postgres` adapter and used Postgres tables for metadata access. When this flag is enabled, the adapter uses the Redshift API (through the Python client) if available, or queries Redshift's `information_schema` tables instead of using the `pg_` tables.

While you shouldn't notice any behavior changes due to this change, however, to be cautious dbt Labs is gating it behind a behavior-change flag and encouraging you to test it before it becoming the default.
23 changes: 20 additions & 3 deletions website/docs/reference/node-selection/state-comparison-caveats.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,16 @@ dbt will mark modified any resource that depends on a changed macro, or on a mac

### Vars

If a model uses a `var` or `env_var` in its definition, dbt is unable today to identify that lineage in such a way that it can include the model in `state:modified` because the `var` or `env_var` value has changed. It's likely that the model will be marked modified if the change in variable results in a different configuration.
<VersionBlock lastVersion="1.8">

If a model uses a `var` or `env_var` in its definition, dbt Core 1.8 and earlier are unable today to identify that lineage in such a way that it can include the model in `state:modified` because the `var` or `env_var` value has changed. It's likely that the model will be marked modified if the change in variable results in a different configuration.
</VersionBlock>

<VersionBlock firstVersion="1.9">

Beginning in dbt Core 1.9, when you set the `state_modified_compare_vars` [behavior flag](/reference/global-configs/behavior-changes#behavior-change-flags) to `True` and a model uses a `var` or `env_var` in its definition, dbt will identify that lineage in such a way that it will include the model in `state:modified` when the `var` or `env_var` value has changed.

</VersionBlock>

### Tests

Expand Down Expand Up @@ -44,12 +53,19 @@ dbt test -s "state:modified" --exclude "test_name:relationships"

### False positives

<VersionBlock firstVersion="1.9">

To reduce false positives during `state:modified` selection due to env-aware logic, you can set the `state_modified_compare_more_unrendered` [behavior flag](/reference/global-configs/behavior-changes#behavior-change-flags) to `True`.

</VersionBlock>

<VersionBlock lastVersion="1.8">
State comparison works by identifying discrepancies between two manifests. Those discrepancies could be the result of:

1. Changes made to a project in development
2. Env-aware logic that causes different behavior based on the `target`, env vars, etc.
2. Env-aware logic that causes different behavior based on the `target`, env vars, etc., which can be avoided if you upgrade to dbt Core 1.9 and set the `state_modified_compare_more_unrendered` [behavior flag](/reference/global-configs/behavior-changes#behavior-change-flags) to `True`.

State comparison detects env-aware config in `dbt_project.yml`. This target-based config registers as a modification:
State comparison detects env-aware config in `dbt_project.yml`. This target-based config won't register as a modification:

<File name='dbt_project.yml'>

Expand All @@ -73,6 +89,7 @@ That means the following config—functionally identical to the snippet above—
materialized = ('table' if target.name == 'prod' else 'view')
) }}
```
</VersionBlock>

### Final note

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ Snowflake's `CREATE ICEBERG TABLE` DDL requires that a `base_location` be provid
#### Base Location Subpath
We recommend using dbt's auto-generated `base_location`. However, if you need to customize the resulting `base_location`, dbt allows users to configure a `base_location_subpath`. When specified, the subpath concatenates to the end of the previously described pattern for `base_location` string generation.

For example, `config(base_location_subpath="prod")` will generate a `base_location` of the form `_dbt/{SCHEMA_NAME}/{MODEL_NAME}`.
For example, `config(base_location_subpath="prod")` will generate a `base_location` of the form `_dbt/{SCHEMA_NAME}/{MODEL_NAME}/prod/`.

A theoretical (but not recommended) use case is re-using an `EXTERNAL VOLUME` while maintaining isolation across development and production environments. We recommend against this as storage permissions should configured on the external volume and underlying storage, not paths that any analytics engineer can modify.

Expand Down
8 changes: 8 additions & 0 deletions website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -1095,6 +1095,14 @@ const sidebarSettings = {
items: [
"reference/global-configs/about-global-configs",
"reference/global-configs/behavior-changes",
{ type: "category",
label: "Adapter behavior changes",
link: { type: "doc", id: "reference/global-configs/adapter-behavior-changes" },
items: [
"reference/global-configs/adapter-behavior-changes",
"reference/global-configs/redshift-changes",
],
},
{
type: "category",
label: "Setting flags",
Expand Down

0 comments on commit 1b34dbe

Please sign in to comment.