diff --git a/website/blog/2022-09-13-the-case-against-cherry-picking.md b/website/blog/2022-09-13-the-case-against-cherry-picking.md index 84a70e08392..580df2be994 100644 --- a/website/blog/2022-09-13-the-case-against-cherry-picking.md +++ b/website/blog/2022-09-13-the-case-against-cherry-picking.md @@ -9,6 +9,10 @@ hide_table_of_contents: false date: 2022-09-13 is_featured: true --- + +:::note You can now use a Staging environment! +This blog post was written before Staging environments. You can now use dbt Cloud can to support the patterns discussed here. Read more about [Staging environments](/docs/deploy/deploy-environments#staging-environment). +::: ## Why do people cherry pick into upper branches? diff --git a/website/blog/2023-11-14-specify-prod-environment.md b/website/blog/2023-11-14-specify-prod-environment.md index c6ad2b31027..0e205abd749 100644 --- a/website/blog/2023-11-14-specify-prod-environment.md +++ b/website/blog/2023-11-14-specify-prod-environment.md @@ -14,6 +14,10 @@ is_featured: false --- +:::note You can now use a Staging environment! +This blog post was written before Staging environments. You can now use dbt Cloud can to support the patterns discussed here. Read more about [Staging environments](/docs/deploy/deploy-environments#staging-environment). +::: + :::tip The Bottom Line: You should [split your Jobs](#how) across Environments in dbt Cloud based on their purposes (e.g. Production and Staging/CI) and set one environment as Production. This will improve your CI experience and enable you to use dbt Explorer. ::: diff --git a/website/docs/docs/cloud/about-cloud-develop-defer.md b/website/docs/docs/cloud/about-cloud-develop-defer.md index 37bfaacfd0c..d5a48ee4654 100644 --- a/website/docs/docs/cloud/about-cloud-develop-defer.md +++ b/website/docs/docs/cloud/about-cloud-develop-defer.md @@ -19,6 +19,8 @@ By default, dbt follows these rules: For a clean slate, it's a good practice to drop the development schema at the start and end of your development cycle. +If you require additional controls over production data, create a [Staging evironment](/docs/deploy/deploy-environments#staging-environment) and dbt will use that, rather than the Production environment, to resolve `{{ ref() }}` functions. + ## Required setup - You must select the **[Production environment](/docs/deploy/deploy-environments#set-as-production-environment)** checkbox in the **Environment Settings** page. @@ -42,7 +44,7 @@ For example, if you were to start developing on a new branch with [nothing in yo One key difference between using `--defer` in the dbt Cloud CLI and the dbt Cloud IDE is that `--defer` is *automatically* enabled in the dbt Cloud CLI for all invocations, compared with production artifacts. You can disable it with the `--no-defer` flag. -The dbt Cloud CLI offers additional flexibility by letting you choose the source environment for deferral artifacts. You can set a `defer-env-id` key in either your `dbt_project.yml` or `dbt_cloud.yml` file. If you do not provide a `defer-env-id` setting, the dbt Cloud CLI will use artifacts from your dbt Cloud environment marked "Production". +The dbt Cloud CLI offers additional flexibility by letting you choose the source environment for deferral artifacts. You can manually set a `defer-env-id` key in either your `dbt_project.yml` or `dbt_cloud.yml` file. By default, the dbt Cloud CLI will prefer metadata from the project's "Staging" environment (if defined), otherwise "Production." diff --git a/website/docs/docs/collaborate/dbt-explorer-faqs.md b/website/docs/docs/collaborate/dbt-explorer-faqs.md index e214a9735de..ef13b697994 100644 --- a/website/docs/docs/collaborate/dbt-explorer-faqs.md +++ b/website/docs/docs/collaborate/dbt-explorer-faqs.md @@ -37,7 +37,9 @@ No. dbt Explorer and all of its features are only available as a dbt Cloud user -dbt Explorer defaults to the latest production state of a project. Support for staging and development (Cloud CLI and IDE) environments is coming soon. Users can only assign a single production and staging environment per dbt Cloud project. +dbt Explorer defaults to the latest production state of a project. Support for staging is now in . + +development (Cloud CLI and IDE) environments is coming soon. Users can only assign one production and one staging environment per dbt Cloud project. diff --git a/website/docs/docs/collaborate/explore-projects.md b/website/docs/docs/collaborate/explore-projects.md index 4633e86d86c..bfeb0284f69 100644 --- a/website/docs/docs/collaborate/explore-projects.md +++ b/website/docs/docs/collaborate/explore-projects.md @@ -230,6 +230,12 @@ Example of the details view for the model `supplies`: +## Staging environment + +dbt Explorer supports views for [Staging deployment environments](/docs/deploy/deploy-environments#staging-environment), in addition to the Production environment. This gives you a unique view into your pre-production data workflows, with the same tools available in production, while providing an extra layer of scrutiny. Once the Staging environment is configured and has a successful job run, it will be visible on the dbt Explorer landing page. + + + ## Related content - [Enterprise permissions](/docs/cloud/manage-access/enterprise-permissions) - [About model governance](/docs/collaborate/govern/about-model-governance) diff --git a/website/docs/docs/collaborate/govern/project-dependencies.md b/website/docs/docs/collaborate/govern/project-dependencies.md index adb4e1f9df8..8265e953839 100644 --- a/website/docs/docs/collaborate/govern/project-dependencies.md +++ b/website/docs/docs/collaborate/govern/project-dependencies.md @@ -35,8 +35,9 @@ In order to add project dependencies and resolve cross-project `ref`, you must: - Use dbt v1.6 or higher for **both** the upstream ("producer") project and the downstream ("consumer") project. - Define models in an upstream ("producer") project that are configured with [`access: public`](/reference/resource-configs/access). To apply the change, rerun a production job. - Have a deployment environment in the upstream ("producer") project [that is set to be your production environment](/docs/deploy/deploy-environments#set-as-production-environment) -- Have a successful run of the upstream ("producer") project -- Have a multi-tenant or single-tenant [dbt Cloud Enterprise](https://www.getdbt.com/pricing) account (Azure ST is not supported but coming soon) +- Have a successful run of the upstream ("producer") project. +- Define and trigger a job before marking the environment as Staging. Read more about [Staging environments with downstream dependencies](/docs/collaborate/govern/project-dependencies#staging-with-downstream-dependencies). +- Have a multi-tenant or single-tenant [dbt Cloud Enterprise](https://www.getdbt.com/pricing) account (Azure ST is not supported but coming soon.) ## Example @@ -104,6 +105,20 @@ with monthly_revenue as ( For more guidance on how to use dbt Mesh, refer to the dedicated [dbt Mesh guide](/best-practices/how-we-mesh/mesh-1-intro). +### Safeguarding production data with staging environments + +When working in a Development environment, cross-project `ref`s normally resolve to the Production environment of the project. However, to protect production data, set up a [Staging deployment environment](/docs/deploy/deploy-environments#staging-environment) within your projects. With a staging environment integrated into the project, any references from external projects during development workflows resolve to the Staging environment. This adds a layer of security between your Deployment and Production environments by limiting access to production data. + +Read [Why use a staging environment](/docs/deploy/deploy-environments#why-use-a-staging-environment) for more information about the benefits. + +#### Staging with downstream dependencies + +dbt Cloud begins using the Staging environment to resolve cross-project references from downstream projects as soon as it exists in a project without "fail-over" to Production. To avoid causing downtime for downstream developers, you should define and trigger a job before marking the environment as Staging: +1. Create a new environment, but do NOT mark it as **Staging**. +2. Define a job in that environment. +3. Trigger the job to run, and ensure it completes successfully. +4. Update the environment to mark it as **Staging**. + ### Comparison If you were to instead install the `jaffle_finance` project as a `package` dependency, you would instead be pulling down its full source code and adding it to your runtime environment. This means: diff --git a/website/docs/docs/dbt-versions/2024-release-notes.md b/website/docs/docs/dbt-versions/2024-release-notes.md index a381d7f49de..31130cee886 100644 --- a/website/docs/docs/dbt-versions/2024-release-notes.md +++ b/website/docs/docs/dbt-versions/2024-release-notes.md @@ -9,17 +9,23 @@ pagination_prev: null dbt Cloud release notes for 2024. +## May 2024 + +- **Behavior change:** Introduced the `require_resource_names_without_spaces` flag, opt-in and disabled by default. If set to `True`, dbt will raise an exception if it finds a resource name containing a space in your project or an installed package. This will become the default in a future version of dbt. Read [No spaces in resource names](/reference/global-configs/legacy-behaviors#no-spaces-in-resource-names) for more information. + ## April 2024 - - You can now set up a continuous deployment (CD) workflow for your projects natively in dbt Cloud. You can now access a beta release of [Merge jobs](/docs/deploy/merge-jobs), which is a new [job type](/docs/deploy/jobs), that enables you to trigger dbt job runs as soon as changes (via Git pull requests) merge into production. + You can now set up a continuous deployment (CD) workflow for your projects natively in dbt Cloud. You can now access a beta release of [Merge jobs](/docs/deploy/merge-jobs), which is a new [job type](/docs/deploy/jobs), that enables you to trigger dbt job runs as soon as changes (via Git pull requests) merge into production. + +- **Behavior change:** Introduced the `require_explicit_package_overrides_for_builtin_materializations` flag, opt-in and disabled by default. If set to `True`, dbt will only use built-in materializations defined in the root project or within dbt, rather than implementations in packages. This will become the default in May 2024 (dbt Core v1.8 and dbt Cloud "Keep on latest version"). Read [Package override for built-in materialization](/reference/global-configs/legacy-behaviors#package-override-for-built-in-materialization) for more information. **dbt Semantic Layer** - **New**: Use Saved selections to [save your query selections](/docs/use-dbt-semantic-layer/gsheets#using-saved-selections) within the [Google Sheets application](/docs/use-dbt-semantic-layer/gsheets). They can be made private or public and refresh upon loading. @@ -41,6 +47,7 @@ dbt Cloud release notes for 2024. - **Fix:** `dbt parse` no longer shows an error when you use a list of filters (instead of just a string filter) on a metric. - **Fix:** `join_to_timespine` now properly gets applied to conversion metric input measures. - **Fix:** Fixed an issue where exports in Redshift were not always committing to the DWH, which also had the side-effect of leaving table locks open. +- **Behavior change:** Introduced the `source_freshness_run_project_hooks` flag, opt-in and disabled by default. If set to `True`, dbt will include `on-run-*` project hooks in the `source freshness` command. This will become the default in a future version of dbt. Read [Project hooks with source freshness](/reference/global-configs/legacy-behaviors#project-hooks-with-source-freshness) for more information. ## February 2024 diff --git a/website/docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md b/website/docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md index caf80840def..bf87434a646 100644 --- a/website/docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md +++ b/website/docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md @@ -4,7 +4,7 @@ id: upgrading-to-v1.8 description: New features and changes in dbt Core v1.8 displayed_sidebar: "docs" --- - + ## Resources - Changelog (coming soon) @@ -31,9 +31,9 @@ Features and functionality new in dbt v1.8. ### New dbt Core adapter installation procedure -Before v1.8, when you installed an adapter, you would automatically get `dbt-core` installed along with the adapter package (if you didn’t already have an existing, compatible version of dbt-core). +Before dbt Core v1.8, whenever you would `pip install` a data warehouse adapter for dbt, `pip` would automatically install `dbt-core` alongside it. The dbt adapter directly depended on components of `dbt-core`, and `dbt-core` depended on the adapter for execution. This bidirectional dependency made it difficult to develop adapters independent of `dbt-core`. -Beginning in v1.8, the [dbt adapters and dbt Core have been decoupled](https://github.com/dbt-labs/dbt-adapters/discussions/87). As a result, you must install _both_ dbt-core and the desired adapter. A new `pip` installation needs to look like this: +Beginning in v1.8, [`dbt-core` and adapters are decoupled](https://github.com/dbt-labs/dbt-adapters/discussions/87). Going forward, your installations should explicitly include _both_ `dbt-core` _and_ the desired adapter. The new `pip` installation command should look like this: ```shell pip install dbt-core dbt-ADAPTER_NAME @@ -44,6 +44,8 @@ For example, you would use the following command if you use Snowflake: pip install dbt-core dbt-snowflake ``` +For the time being, we have maintained install-time dependencies to avoid breaking existing scripts in surprising ways; `pip install dbt-snowflake` will continue to install the latest versions of both `dbt-core` and 1dbt-snowflake`. Given that we may remove this implicit dependency in future versions, we strongly encourage you to update install scripts **now**. + ### Unit Tests Historically, dbt's test coverage was confined to [“data” tests](/docs/build/data-tests), assessing the quality of input data or resulting datasets' structure. @@ -78,23 +80,23 @@ models: ``` -### The `--empty` flag +#### The `--empty` flag The [`run`](/reference/commands/run#the-`--empty`-flag) and [`build`](/reference/commands/build#the---empty-flag) commands now support the `--empty` flag for building schema-only dry runs. The `--empty` flag limits the refs and sources to zero rows. dbt will still execute the model SQL against the target data warehouse but will avoid expensive reads of input data. This validates dependencies and ensures your models will build properly. -### Spaces in dbt model names +### Managing changes to legacy behaviors + +dbt Core v1.8 has introduced a flags has been created for [managing changes to legacy behaviors](/reference/global-configs/legacy-behaviors). You may opt into recently introduced changes (disabled by default), or opt out of mature changes (enabled by default), by setting `True` / `False` values, respectively, for `flags` in `dbt_project.yml`. -We will begin deprecating support for spaces in dbt model names in v1.8 (raising a warning) before removing support entirely in v1.9 (raising an error). Reasons for removing spaces in model names include: -- Spaces in a model name make it impossible to `--select` the model name because the argument gets split into pieces over spaces very early in the pipeline. -- Most warehouses do not accept a table, or other object, with a space in its name. +You can read more about each of these behavior changes in the following links: -To upgrade, replace any spaces in the model file name with an underscore and update any associated YAML that contains the model name to match. You can keep spaces in the database table name by configuring a [custom `alias`](/docs/build/custom-aliases#usage). +- (Mature, enabled by default) [Require explicit package overrides for builtin materializations](/reference/global-configs/legacy-behaviors#require_explicit_package_overrides_for_builtin_materializations) +- (Introduced, disabled by default) [Require resource names without spaces](https://docs.getdbt.com/reference/global-configs/legacy-behaviors#require_resource_names_without_spaces) +- (Introduced, disabled by default) [Run project hooks (`on-run-*`) in the `dbt source freshness` command](/reference/global-configs/legacy-behaviors#source_freshness_run_project_hooks) ## Quick hits -- [Global config flags](/reference/global-configs/about-global-configs) are deprecated from the [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml) file and should be moved to the [`dbt_project.yml`](/reference/dbt_project.yml). -- A new subcategory of flags has been created for [legacy behaviors](/reference/global-configs/legacy-behaviors). -- The [`--indirect_selection`](/reference/global-configs/indirect-selection) flag used with `dbt test` or `dbt build` configures which tests to run for the nodes you specify. +- Custom defaults of [global config flags](/reference/global-configs/about-global-configs) should be set in the `flags` dictionary in [`dbt_project.yml`](/reference/dbt_project.yml), instead of in [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml). Support for `profiles.yml` has been deprecated. - New CLI flag [`--resource-type`/`--exclude-resource-type`](/reference/global-configs/resource-type) for including/excluding resources from dbt `build`, `run`, and `clone`. - To improve performance, dbt now issues a single (batch) query when calculating `source freshness` through metadata, instead of executing a query per source. - Syntax for `DBT_ENV_SECRET_` has changed to `DBT_ENV_SECRET` and no longer requires the closing underscore. diff --git a/website/docs/docs/deploy/deploy-environments.md b/website/docs/docs/deploy/deploy-environments.md index e42d39bad82..42f34740164 100644 --- a/website/docs/docs/deploy/deploy-environments.md +++ b/website/docs/docs/deploy/deploy-environments.md @@ -18,7 +18,7 @@ To learn different approaches to managing dbt Cloud environments and recommendat Learn more about development vs. deployment environments in [dbt Cloud Environments](/docs/dbt-cloud-environments). -There are three types of deployment environments that serve different needs: +There are three types of deployment environments: - **Production**: Environment for transforming data and building pipelines for production use. - **Staging**: Environment for working with production tools while limiting access to production data. - **General**: General use environment for deployment development. @@ -41,6 +41,46 @@ In dbt Cloud, each project can have one designated deployment environment, which For Semantic Layer-eligible customers, the next section of environment settings is the Semantic Layer configurations. [The Semantic Layer setup guide](/docs/use-dbt-semantic-layer/setup-sl) has the most up-to-date setup instructions! +## Staging environment + +:::note +Currently in limited availability beta. Contact support or your account team if you're interested in beta access. +::: + +Use a Staging environment to grant developers access to deployment workflows and tools while controlling access to production data. You can do this in a couple of ways, but the most straightforward is to configure Staging with a long-living branch (for example, `staging`) similar to but separate from the primary branch (for example, `main`). + +In this scenario, the workflows would ideally move upstream from the Development environment -> Staging environment -> Production environment with developer branches feeding into the `staging` branch, then ultimately merging into `main`. In many cases, the `main` and `staging` branches will be identical after a merge and remain until the next batch of changes from the `development` branches are ready to be elevated. We recommend setting branch protection rules on `staging` similar to `main`. + +### Why use a staging environment + +There are two primary motivations for using a Staging environment: +1. An additional validation layer before changes are deployed into Production. You can deploy, test, and explore your dbt models in Staging. +2. Clear isolation between development workflows and production data. It enables developers to work in metadata-powered ways, using features like deferral and cross-project references, without accessing data in production deployments. + +:::info Coming soon: environment-level permissions +Provide developers with the ability to create, edit, and trigger ad hoc jobs in the Staging environment, while keeping the Production environment locked down. +::: + +Let's say you have `Project B` downstream of `Project A` with cross-project refs configured in the models. When developers work in the IDE for `Project B`, cross-project refs will resolve to the Staging environment of `Project A`, rather than production. You'll get the same results with those refs when jobs are run in the Staging environment. Only the Production environment will reference the Production data, keeping the data and access isolated without needing separate projects. + +If `Project B` also has a Staging deployment, then references to unbuilt upstream models within `Project B` will resolve to that environment, using [deferral](/docs/cloud/about-cloud-develop-defer), rather than resolving to the models in Production. This saves developers time and warehouse spend, while preserving clear separation of environments. + +Finally, the Staging environment has its own view in [dbt Explorer](/docs/collaborate/explore-projects), giving you a full view of your prod and pre-prod data. + + + + +### Create a Staging environment + +In the dbt Cloud, navigate to **Deploy** -> **Environments** and then click **Create Environment**. Select **Deployment** as the environment type. The option will be greyed out if you already have a development environment. + + + + +Follow the steps outlined in [deployment credentials](#deployment-connection) to complete the remainder of the environment setup. + +We recommend that the data warehouse credentials be for a dedicated user or service principal. + ## Deployment connection @@ -189,47 +229,6 @@ This section allows you to determine the credentials that should be used when co - -## Staging environment - -:::note -Currently in limited availability beta. Contact support or your account team if you're interested in beta access. -::: - -Staging environments are useful ways to grant developers access to deployment workflows and tools while controlling access to production data. They are configured with their own long-living branch (for example, `staging`) that may be very similar to `main` in many ways while potentially limiting the data the developers can access. - -Ideally, the workflows would move upstream from the Development environment -> Staging environment -> Production environment with developer branches feeding into the staging branch, then ultimately `main`. In many cases, the `main` and `staging` branches will be identical after a merge and remain until the next batch of changes from the `development` branches are ready to be elevated. We recommend setting branch protection rules on `staging` similar to `main`. - -### Create a staging environment - -In the dbt Cloud, navigate to **Deploy** -> **Environments** and then click **Create Environment**. Select **Deployment** as the environment type. The option will be greyed out if you already have a development environment. - - - - -Follow the steps outlined in [deployment credentials](#deployment-connection) to complete the remainder of the environment setup. - -We recommend that the data warehouse credentials be for a dedicated user or service principal. - -### Why use a staging environment - -There are two primary motivations for using a Staging environment: -1. An additional validation layer before changes are deployed into Production. You can deploy, test, and explore your dbt models in Staging. -2. Clear isolation between development workflows and production data. It enables developers to work in metadata-powered ways, using features like deferral and cross-project references, without accessing data in production deployments. - -:::info Coming soon: environment-level permissions -Provide developers with the ability to create, edit, and trigger ad hoc jobs in the Staging environment, while keeping the Production environment locked down. -::: - -Let's say you have `Project B` downstream of `Project A` with cross-project refs configured in the models. When developers work in the IDE for `Project B`, cross-project refs will resolve to the Staging environment of `Project A`, rather than production. You'll get the same results with those refs when jobs are run in the Staging environment. Only the Production environment will reference the Production data, keeping the data and access isolated without needing separate projects. - -If `Project B` also has a Staging deployment, then references to unbuilt upstream models within `Project B` will resolve to that environment, using [deferral](/docs/cloud/about-cloud-develop-defer), rather than resolving to the models in Production. This saves developers time and warehouse spend, while preserving clear separation of environments. - -Finally, the Staging environment has its own view in [dbt Explorer](/docs/collaborate/explore-projects), giving you a full view of your prod and pre-prod data. - - - - ## Related docs - [dbt Cloud environment best practices](/guides/set-up-ci) diff --git a/website/docs/reference/global-configs/legacy-behaviors.md b/website/docs/reference/global-configs/legacy-behaviors.md index d1ef606b149..cd898d55187 100644 --- a/website/docs/reference/global-configs/legacy-behaviors.md +++ b/website/docs/reference/global-configs/legacy-behaviors.md @@ -12,15 +12,82 @@ Another category of flags provides existing projects with a migration window for - Providing maintainability of dbt software. Every fork in behavior requires additional testing & cognitive overhead that slows future development. These flags exist to facilitate migration from "current" to "better," not to stick around forever. These flags go through three phases of development: -1. **Introduction:** The flag is introduced, and the logic added within dbt to support both 'old' + 'new' behaviours. -2. **Maturity:** The default value of the flag is switched, from 'old' to 'new.' -3. **Deprecation:** The flag, having been marked for deprecation (with user warnings), is removed. The logic to support the ‘old’ behaviour is removed from dbt codebases. +1. **Introduction (disabled by default):** dbt adds logic to support both 'old' + 'new' behaviors. The 'new' behavior is gated behind a flag, disabled by default, preserving the old behavior. +2. **Maturity (enabled by default):** The default value of the flag is switched, from `false` to `true`, enabling the new behavior by default. Users can preserve the 'old' behavior and opt out of the 'new' behavior by setting the flag to `false` in their projects. They may see deprecation warnings when they do so. +3. **Removal (generally enabled):** After marking the flag for deprecation, we remove it along with the 'old' behavior it supported from the dbt codebases. We aim to support most flags indefinitely, but we're not committed to supporting them forever. If we choose to remove a flag, we'll offer significant advance notice. -These flags can _only_ be set in `dbt_project.yml`. They configure behaviors closely tied to project code. They should be defined in version control, and changed via pull/merge request, with the same testing and peer review. +## Behavior change flags -## Behaviors +These flags _must_ be set in the `flags` dictionary in `dbt_project.yml`. They configure behaviors closely tied to project code, which means they should be defined in version control and modified through pull or merge requests, with the same testing and peer review. -### `source_freshness_run_project_hooks` +The following example displays the current flags and their current default values in the latest dbt Cloud and dbt Core versions. To opt out of a specific behavior change, set the values of the flag to `False` in `dbt_project.yml`. You'll continue to see warnings for legacy behaviors that you have opted out of explicitly until you either resolve them (switch the flag to `True`) or choose to silence the warnings using the `warn_error_options.silence` flag. + + + +```yml +flags: + require_explicit_package_overrides_for_builtin_materializations: False + require_model_names_without_spaces: False + source_freshness_run_project_hooks: False +``` + + + +When we use dbt Cloud in the following table, we're referring to "[Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version)." + +| Flag | dbt Cloud: Intro | dbt Cloud: Maturity | dbt Core: Intro | dbt Core: Maturity | +|-----------------------------------------------------------------|------------------|---------------------|-----------------|--------------------| +| require_explicit_package_overrides_for_builtin_materializations | 2024.04.141 | 2024.05.TBD | 1.6.14, 1.7.14 | 1.8.0 | +| require_resource_names_without_spaces | 2024.05.TBD | 2024.06.TBD | 1.8.0 | 1.9.0 | +| source_freshness_run_project_hooks | 2024.03.61 | 2024.06.TBD | 1.8.0 | 1.9.0 | + + + +### Package override for built-in materialization + +Setting the `require_explicit_package_overrides_for_builtin_materializations` flag to `True` prevents this automatic override. + +We have deprecated the behavior where installed packages could override built-in materializations without your explicit opt-in. When this flag is set to `True`, a materialization defined in a package that matches the name of a built-in materialization will no longer be included in the search and resolution order. Unlike macros, materializations don't use the `search_order` defined in the project `dispatch` config. + +The built-in materializations are `'view'`, `'table'`, `'incremental'`, `'materialized_view'` for models as well as `'test'`, `'unit'`, `'snapshot'`, `'seed'`, and `'clone'`. + +You can still explicitly override built-in materializations, in favor of a materialization defined in a package, by reimplementing the built-in materialization in your root project and wrapping the package implementation. + + + +```sql +{% materialization view, snowflake %} + {{ return(my_installed_package_name.materialization_view_snowflake()) }} +{% endmaterialization %} +``` + + + +In the future, we may extend the project-level [`dispatch` configuration](/reference/project-configs/dispatch-config) to support a list of authorized packages for overriding built-in materialization. + + + +The following flags were introduced in a future version of dbt Core. If you're still using an older version, then you have the legacy behavior by default (when each flag is `False`). + + + +### No spaces in resource names + +The `require_resource_names_without_spaces` flag enforces using resource names without spaces. + +The names of dbt resources (models, sources, etc) should contain letters, numbers, and underscores. We highly discourage the use of other characters, especially spaces. To that end, we have deprecated support for spaces in resource names. When the `require_resource_names_without_spaces` flag is set to `True`, dbt will raise an exception (instead of a deprecation warning) if it detects a space in a resource name. + + + +```sql +-- This model file should be renamed to model_name_with_underscores.sql +``` + + + +### Project hooks with source freshness + +Set the `source_freshness_run_project_hooks` flag to `True` to include "project hooks" ([`on-run-start` / `on-run-end`](/reference/project-configs/on-run-start-on-run-end)) in the `dbt source freshness` command execution. If you have specific project [`on-run-start` / `on-run-end`](/reference/project-configs/on-run-start-on-run-end) hooks that should not run before/after `source freshness` command, you can add a conditional check to those hooks: @@ -31,7 +98,3 @@ on-run-start: - '{{ ... if flags.WHICH != 'freshness' }}' ``` - -You can configure the `dbt source freshness` command to include on-run-start / on-run-end hooks in v1.8 with a planned removal in v1.9. The default value is `True`. - -In the meantime, you can set `source_freshness_run_project_hooks: False` to preserve the legacy behavior.