Skip to content

Commit

Permalink
Revise docs for behavior change flags (#5389)
Browse files Browse the repository at this point in the history 
## What are you changing in this pull request and why?

**Legacy behaviors:**
- Update explanations
- Rename behavior change flags
- Add table with dates for Intro / Maturity

**v1.8 guide:**
- Reorganize + update with links to the legacy behavior page

**Depends on changes tracked here:**
- dbt-labs/dbt-core#10062

---------

Co-authored-by: Mirna Wong <[email protected]>
Co-authored-by: Leona B. Campbell <[email protected]>
  • Loading branch information
@jtcohen6 @mirnawong1 @runleonarun
3 people authored May 2, 2024
1 parent 2c08d72 commit d8f0ef2
Show file tree
Hide file tree
Showing 3 changed files with 95 additions and 23 deletions.
9 changes: 8 additions & 1 deletion website/docs/docs/dbt-versions/2024-release-notes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,23 @@ pagination_prev: null

dbt Cloud release notes for 2024.

## May 2024

- **Behavior change:** Introduced the `require_resource_names_without_spaces` flag, opt-in and disabled by default. If set to `True`, dbt will raise an exception if it finds a resource name containing a space in your project or an installed package. This will become the default in a future version of dbt. Read [No spaces in resource names](/reference/global-configs/legacy-behaviors#no-spaces-in-resource-names) for more information.

## April 2024

- <expandable alt_header="New: Merge jobs" >

<Lifecycle status="beta" />

You can now set up a continuous deployment (CD) workflow for your projects natively in dbt Cloud. You can now access a beta release of [Merge jobs](/docs/deploy/merge-jobs), which is a new [job type](/docs/deploy/jobs), that enables you to trigger dbt job runs as soon as changes (via Git pull requests) merge into production.
You can now set up a continuous deployment (CD) workflow for your projects natively in dbt Cloud. You can now access a beta release of [Merge jobs](/docs/deploy/merge-jobs), which is a new [job type](/docs/deploy/jobs), that enables you to trigger dbt job runs as soon as changes (via Git pull requests) merge into production.

<Lightbox src="/img/docs/dbt-cloud/using-dbt-cloud/example-create-merge-job.png" width="90%" title="Example of creating a merge job"/>

</expandable>

- **Behavior change:** Introduced the `require_explicit_package_overrides_for_builtin_materializations` flag, opt-in and disabled by default. If set to `True`, dbt will only use built-in materializations defined in the root project or within dbt, rather than implementations in packages. This will become the default in May 2024 (dbt Core v1.8 and dbt Cloud "Keep on latest version"). Read [Package override for built-in materialization](/reference/global-configs/legacy-behaviors#package-override-for-built-in-materialization) for more information.

**dbt Semantic Layer**
- **New**: Use Saved selections to [save your query selections](/docs/use-dbt-semantic-layer/gsheets#using-saved-selections) within the [Google Sheets application](/docs/use-dbt-semantic-layer/gsheets). They can be made private or public and refresh upon loading.
Expand All @@ -41,6 +47,7 @@ dbt Cloud release notes for 2024.
- **Fix:** `dbt parse` no longer shows an error when you use a list of filters (instead of just a string filter) on a metric.
- **Fix:** `join_to_timespine` now properly gets applied to conversion metric input measures.
- **Fix:** Fixed an issue where exports in Redshift were not always committing to the DWH, which also had the side-effect of leaving table locks open.
- **Behavior change:** Introduced the `source_freshness_run_project_hooks` flag, opt-in and disabled by default. If set to `True`, dbt will include `on-run-*` project hooks in the `source freshness` command. This will become the default in a future version of dbt. Read [Project hooks with source freshness](/reference/global-configs/legacy-behaviors#project-hooks-with-source-freshness) for more information.


## February 2024
Expand Down
26 changes: 14 additions & 12 deletions website/docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ id: upgrading-to-v1.8
description: New features and changes in dbt Core v1.8
displayed_sidebar: "docs"
---

## Resources

- Changelog (coming soon)
Expand All @@ -31,9 +31,9 @@ Features and functionality new in dbt v1.8.

### New dbt Core adapter installation procedure

Before v1.8, when you installed an adapter, you would automatically get `dbt-core` installed along with the adapter package (if you didn’t already have an existing, compatible version of dbt-core).
Before dbt Core v1.8, whenever you would `pip install` a data warehouse adapter for dbt, `pip` would automatically install `dbt-core` alongside it. The dbt adapter directly depended on components of `dbt-core`, and `dbt-core` depended on the adapter for execution. This bidirectional dependency made it difficult to develop adapters independent of `dbt-core`.

Beginning in v1.8, the [dbt adapters and dbt Core have been decoupled](https://github.com/dbt-labs/dbt-adapters/discussions/87). As a result, you must install _both_ dbt-core and the desired adapter. A new `pip` installation needs to look like this:
Beginning in v1.8, [`dbt-core` and adapters are decoupled](https://github.com/dbt-labs/dbt-adapters/discussions/87). Going forward, your installations should explicitly include _both_ `dbt-core` _and_ the desired adapter. The new `pip` installation command should look like this:

```shell
pip install dbt-core dbt-ADAPTER_NAME
Expand All @@ -44,6 +44,8 @@ For example, you would use the following command if you use Snowflake:
pip install dbt-core dbt-snowflake
```

For the time being, we have maintained install-time dependencies to avoid breaking existing scripts in surprising ways; `pip install dbt-snowflake` will continue to install the latest versions of both `dbt-core` and 1dbt-snowflake`. Given that we may remove this implicit dependency in future versions, we strongly encourage you to update install scripts **now**.

### Unit Tests

Historically, dbt's test coverage was confined to [“data” tests](/docs/build/data-tests), assessing the quality of input data or resulting datasets' structure.
Expand Down Expand Up @@ -78,23 +80,23 @@ models:

```

### The `--empty` flag
#### The `--empty` flag

The [`run`](/reference/commands/run#the-`--empty`-flag) and [`build`](/reference/commands/build#the---empty-flag) commands now support the `--empty` flag for building schema-only dry runs. The `--empty` flag limits the refs and sources to zero rows. dbt will still execute the model SQL against the target data warehouse but will avoid expensive reads of input data. This validates dependencies and ensures your models will build properly.

### Spaces in dbt model names
### Managing changes to legacy behaviors

dbt Core v1.8 has introduced a flags has been created for [managing changes to legacy behaviors](/reference/global-configs/legacy-behaviors). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `True` / `False` values, respectively, for `flags` in `dbt_project.yml`.

We will begin deprecating support for spaces in dbt model names in v1.8 (raising a warning) before removing support entirely in v1.9 (raising an error). Reasons for removing spaces in model names include:
- Spaces in a model name make it impossible to `--select` the model name because the argument gets split into pieces over spaces very early in the pipeline.
- Most warehouses do not accept a table, or other object, with a space in its name.
You can read more about each of these behavior changes in the following links:

To upgrade, replace any spaces in the model file name with an underscore and update any associated YAML that contains the model name to match. You can keep spaces in the database table name by configuring a [custom `alias`](/docs/build/custom-aliases#usage).
- (Mature, enabled by default) [Require explicit package overrides for builtin materializations](/reference/global-configs/legacy-behaviors#require_explicit_package_overrides_for_builtin_materializations)
- (Introduced, disabled by default) [Require resource names without spaces](https://docs.getdbt.com/reference/global-configs/legacy-behaviors#require_resource_names_without_spaces)
- (Introduced, disabled by default) [Run project hooks (`on-run-*`) in the `dbt source freshness` command](/reference/global-configs/legacy-behaviors#source_freshness_run_project_hooks)

## Quick hits

- [Global config flags](/reference/global-configs/about-global-configs) are deprecated from the [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml) file and should be moved to the [`dbt_project.yml`](/reference/dbt_project.yml).
- A new subcategory of flags has been created for [legacy behaviors](/reference/global-configs/legacy-behaviors).
- The [`--indirect_selection`](/reference/global-configs/indirect-selection) flag used with `dbt test` or `dbt build` configures which tests to run for the nodes you specify.
- Custom defaults of [global config flags](/reference/global-configs/about-global-configs) should be set in the `flags` dictionary in [`dbt_project.yml`](/reference/dbt_project.yml), instead of in [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml). Support for `profiles.yml` has been deprecated.
- New CLI flag [`--resource-type`/`--exclude-resource-type`](/reference/global-configs/resource-type) for including/excluding resources from dbt `build`, `run`, and `clone`.
- To improve performance, dbt now issues a single (batch) query when calculating `source freshness` through metadata, instead of executing a query per source.
- Syntax for `DBT_ENV_SECRET_` has changed to `DBT_ENV_SECRET` and no longer requires the closing underscore.
Expand Down
83 changes: 73 additions & 10 deletions website/docs/reference/global-configs/legacy-behaviors.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,82 @@ Another category of flags provides existing projects with a migration window for
- Providing maintainability of dbt software. Every fork in behavior requires additional testing & cognitive overhead that slows future development. These flags exist to facilitate migration from "current" to "better," not to stick around forever.

These flags go through three phases of development:
1. **Introduction:** The flag is introduced, and the logic added within dbt to support both 'old' + 'new' behaviours.
2. **Maturity:** The default value of the flag is switched, from 'old' to 'new.'
3. **Deprecation:** The flag, having been marked for deprecation (with user warnings), is removed. The logic to support the ‘old’ behaviour is removed from dbt codebases.
1. **Introduction (disabled by default):** dbt adds logic to support both 'old' + 'new' behaviors. The 'new' behavior is gated behind a flag, disabled by default, preserving the old behavior.
2. **Maturity (enabled by default):** The default value of the flag is switched, from `false` to `true`, enabling the new behavior by default. Users can preserve the 'old' behavior and opt out of the 'new' behavior by setting the flag to `false` in their projects. They may see deprecation warnings when they do so.
3. **Removal (generally enabled):** After marking the flag for deprecation, we remove it along with the 'old' behavior it supported from the dbt codebases. We aim to support most flags indefinitely, but we're not committed to supporting them forever. If we choose to remove a flag, we'll offer significant advance notice.

These flags can _only_ be set in `dbt_project.yml`. They configure behaviors closely tied to project code. They should be defined in version control, and changed via pull/merge request, with the same testing and peer review.
## Behavior change flags

## Behaviors
These flags _must_ be set in the `flags` dictionary in `dbt_project.yml`. They configure behaviors closely tied to project code, which means they should be defined in version control and modified through pull or merge requests, with the same testing and peer review.

### `source_freshness_run_project_hooks`
The following example displays the current flags and their current default values in the latest dbt Cloud and dbt Core versions. To opt out of a specific behavior change, set the values of the flag to `False` in `dbt_project.yml`. You'll continue to see warnings for legacy behaviors that you have opted out of explicitly until you either resolve them (switch the flag to `True`) or choose to silence the warnings using the `warn_error_options.silence` flag.

<File name='dbt_project.yml'>

```yml
flags:
require_explicit_package_overrides_for_builtin_materializations: False
require_model_names_without_spaces: False
source_freshness_run_project_hooks: False
```
</File>
When we use dbt Cloud in the following table, we're referring to "[Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version)."
| Flag | dbt Cloud: Intro | dbt Cloud: Maturity | dbt Core: Intro | dbt Core: Maturity |
|-----------------------------------------------------------------|------------------|---------------------|-----------------|--------------------|
| require_explicit_package_overrides_for_builtin_materializations | 2024.04.141 | 2024.05.TBD | 1.6.14, 1.7.14 | 1.8.0 |
| require_resource_names_without_spaces | 2024.05.TBD | 2024.06.TBD | 1.8.0 | 1.9.0 |
| source_freshness_run_project_hooks | 2024.03.61 | 2024.06.TBD | 1.8.0 | 1.9.0 |
### Package override for built-in materialization
Setting the `require_explicit_package_overrides_for_builtin_materializations` flag to `True` prevents this automatic override.

We have deprecated the behavior where installed packages could override built-in materializations without your explicit opt-in. When this flag is set to `True`, a materialization defined in a package that matches the name of a built-in materialization will no longer be included in the search and resolution order. Unlike macros, materializations don't use the `search_order` defined in the project `dispatch` config.

The built-in materializations are `'view'`, `'table'`, `'incremental'`, `'materialized_view'` for models as well as `'test'`, `'unit'`, `'snapshot'`, `'seed'`, and `'clone'`.

You can still explicitly override built-in materializations, in favor of a materialization defined in a package, by reimplementing the built-in materialization in your root project and wrapping the package implementation.

<File name='macros/materialization_view.sql'>

```sql
{% materialization view, snowflake %}
{{ return(my_installed_package_name.materialization_view_snowflake()) }}
{% endmaterialization %}
```

</File>

In the future, we may extend the project-level [`dispatch` configuration](/reference/project-configs/dispatch-config) to support a list of authorized packages for overriding built-in materialization.

<VersionBlock lastVersion="1.7">

The following flags were introduced in a future version of dbt Core. If you're still using an older version, then you have the legacy behavior by default (when each flag is `False`).

</VersionBlock>

### No spaces in resource names

The `require_resource_names_without_spaces` flag enforces using resource names without spaces.

The names of dbt resources (models, sources, etc) should contain letters, numbers, and underscores. We highly discourage the use of other characters, especially spaces. To that end, we have deprecated support for spaces in resource names. When the `require_resource_names_without_spaces` flag is set to `True`, dbt will raise an exception (instead of a deprecation warning) if it detects a space in a resource name.

<File name='models/model name with spaces.sql'>

```sql
-- This model file should be renamed to model_name_with_underscores.sql
```

</File>

### Project hooks with source freshness

Set the `source_freshness_run_project_hooks` flag to `True` to include "project hooks" ([`on-run-start` / `on-run-end`](/reference/project-configs/on-run-start-on-run-end)) in the `dbt source freshness` command execution.

If you have specific project [`on-run-start` / `on-run-end`](/reference/project-configs/on-run-start-on-run-end) hooks that should not run before/after `source freshness` command, you can add a conditional check to those hooks:

Expand All @@ -31,7 +98,3 @@ on-run-start:
- '{{ ... if flags.WHICH != 'freshness' }}'
```
</File>
You can configure the `dbt source freshness` command to include on-run-start / on-run-end hooks in v1.8 with a planned removal in v1.9. The default value is `True`.

In the meantime, you can set `source_freshness_run_project_hooks: False` to preserve the legacy behavior.

0 comments on commit d8f0ef2

Please sign in to comment.