Staging environment adds

website/blog/2022-09-13-the-case-against-cherry-picking.md

@@ -9,6 +9,10 @@ hide_table_of_contents: false
 date: 2022-09-13
 is_featured: true
 ---
+
+:::note You can now use a Staging environment!
+This blog post was written before Staging environments. You can now use dbt Cloud can to support the patterns discussed here. Read more about [Staging environments](/docs/deploy/deploy-environments#staging-environment).
+:::
 
 ## Why do people cherry pick into upper branches?
 

website/blog/2023-11-14-specify-prod-environment.md

@@ -14,6 +14,10 @@ is_featured: false
 
 ---
 
+:::note You can now use a Staging environment!
+This blog post was written before Staging environments. You can now use dbt Cloud can to support the patterns discussed here. Read more about [Staging environments](/docs/deploy/deploy-environments#staging-environment).
+:::
+
 :::tip The Bottom Line:
 You should [split your Jobs](#how) across Environments in dbt Cloud based on their purposes (e.g. Production and Staging/CI) and set one environment as Production. This will improve your CI experience and enable you to use dbt Explorer.
 :::

website/docs/docs/cloud/about-cloud-develop-defer.md

@@ -19,6 +19,8 @@ By default, dbt follows these rules:
 
 For a clean slate, it's a good practice to drop the development schema at the start and end of your development cycle.
 
+If you require additional controls over production data, create a [Staging evironment](/docs/deploy/deploy-environments#staging-environment) <Lifecycle status='beta' /> and dbt will use that, rather than the Production environment, to resolve `{{ ref() }}` functions.
+
 ## Required setup
 
 - You must select the **[Production environment](/docs/deploy/deploy-environments#set-as-production-environment)** checkbox in the **Environment Settings** page. 
@@ -42,7 +44,7 @@ For example, if you were to start developing on a new branch with [nothing in yo
 
 One key difference between using `--defer` in the dbt Cloud CLI and the dbt Cloud IDE is that `--defer` is *automatically* enabled in the dbt Cloud CLI for all invocations, compared with production artifacts. You can disable it with the `--no-defer` flag.
 
-The dbt Cloud CLI offers additional flexibility by letting you choose the source environment for deferral artifacts. You can set a `defer-env-id` key in either your `dbt_project.yml` or `dbt_cloud.yml` file. If you do not provide a `defer-env-id` setting, the dbt Cloud CLI will use artifacts from your dbt Cloud environment marked "Production". 
+The dbt Cloud CLI offers additional flexibility by letting you choose the source environment for deferral artifacts. You can manually set a `defer-env-id` key in either your `dbt_project.yml` or `dbt_cloud.yml` file. By default, the dbt Cloud CLI will prefer metadata from the project's "Staging" environment (if defined), otherwise "Production."
 
 <File name="dbt_cloud.yml">
 

website/docs/docs/collaborate/dbt-explorer-faqs.md

@@ -37,7 +37,9 @@ No. dbt Explorer and all of its features are only available as a dbt Cloud user
 
 <expandable alt_header="How does dbt Explorer support dbt Cloud environments?" >
 
-dbt Explorer defaults to the latest production state of a project. Support for staging and development (Cloud CLI and IDE) environments is coming soon. Users can only assign a single production and staging environment per dbt Cloud project. 
+dbt Explorer defaults to the latest production state of a project. Support for staging is now in <Lifecycle status='beta' />. 
+
+development (Cloud CLI and IDE) environments is coming soon. Users can only assign one production and one staging environment per dbt Cloud project. 
 
 </expandable>
 

website/docs/docs/collaborate/explore-projects.md

@@ -230,6 +230,12 @@ Example of the details view for the model `supplies`:
 
 <Lightbox src="/img/docs/collaborate/dbt-explorer/example-model-details.png" width="100%" title="Example of resource details" />
 
+## Staging environment <Lifecycle status='beta' />
+
+dbt Explorer supports views for [Staging deployment environments](/docs/deploy/deploy-environments#staging-environment), in addition to the Production environment. This gives you a unique view into your pre-production data workflows, with the same tools available in production, while providing an extra layer of scrutiny. Once the Staging environment is configured and has a successful job run, it will be visible on the dbt Explorer landing page.
+
+<Lightbox src="/img/docs/collaborate/dbt-explorer/explore-staging-env.png" width="85%" title="Explore in a staging environment" />
+
 ## Related content
 - [Enterprise permissions](/docs/cloud/manage-access/enterprise-permissions) 
 - [About model governance](/docs/collaborate/govern/about-model-governance)

website/docs/docs/collaborate/govern/project-dependencies.md

@@ -35,8 +35,9 @@ In order to add project dependencies and resolve cross-project `ref`, you must:
 - Use dbt v1.6 or higher for **both** the upstream ("producer") project and the downstream ("consumer") project.
 - Define models in an upstream ("producer") project that are configured with [`access: public`](/reference/resource-configs/access). To apply the change, rerun a production job.
 - Have a deployment environment in the upstream ("producer") project [that is set to be your production environment](/docs/deploy/deploy-environments#set-as-production-environment)
-- Have a successful run of the upstream ("producer") project
-- Have a multi-tenant or single-tenant [dbt Cloud Enterprise](https://www.getdbt.com/pricing) account (Azure ST is not supported but coming soon) 
+- Have a successful run of the upstream ("producer") project.
+- Define and trigger a job before marking the environment as Staging. Read more about [Staging environments with downstream dependencies](/docs/collaborate/govern/project-dependencies#staging-with-downstream-dependencies).
+- Have a multi-tenant or single-tenant [dbt Cloud Enterprise](https://www.getdbt.com/pricing) account (Azure ST is not supported but coming soon.)
 
 ## Example
 
@@ -104,6 +105,20 @@ with monthly_revenue as (
 
 For more guidance on how to use dbt Mesh, refer to the dedicated [dbt Mesh guide](/best-practices/how-we-mesh/mesh-1-intro).
 
+### Safeguarding production data with staging environments
+
+When working in a Development environment, cross-project `ref`s normally resolve to the Production environment of the project. However, to protect production data, set up a [Staging deployment environment](/docs/deploy/deploy-environments#staging-environment) within your projects. With a staging environment integrated into the project, any references from external projects during development workflows resolve to the Staging environment. This adds a layer of security between your Deployment and Production environments by limiting access to production data.
+
+Read [Why use a staging environment](/docs/deploy/deploy-environments#why-use-a-staging-environment) for more information about the benefits. 
+
+#### Staging with downstream dependencies
+
+dbt Cloud begins using the Staging environment to resolve cross-project references from downstream projects as soon as it exists in a project without "fail-over" to Production. To avoid causing downtime for downstream developers, you should define and trigger a job before marking the environment as Staging:
+1. Create a new environment, but do NOT mark it as **Staging**.
+2. Define a job in that environment.
+3. Trigger the job to run, and ensure it completes successfully.
+4. Update the environment to mark it as **Staging**.
+
 ### Comparison
 
 If you were to instead install the `jaffle_finance` project as a `package` dependency, you would instead be pulling down its full source code and adding it to your runtime environment. This means:

website/docs/docs/deploy/deploy-environments.md

@@ -18,7 +18,7 @@ To learn different approaches to managing dbt Cloud environments and recommendat
 
 Learn more about development vs. deployment environments in [dbt Cloud Environments](/docs/dbt-cloud-environments).
 
-There are three types of deployment environments that serve different needs:
+There are three types of deployment environments:
 - **Production**: Environment for transforming data and building pipelines for production use.
 - **Staging**<Lifecycle status='beta' />: Environment for working with production tools while limiting access to production data.
 - **General**: General use environment for deployment development. 
@@ -41,6 +41,46 @@ In dbt Cloud, each project can have one designated deployment environment, which
 
 For Semantic Layer-eligible customers, the next section of environment settings is the Semantic Layer configurations. [The Semantic Layer setup guide](/docs/use-dbt-semantic-layer/setup-sl) has the most up-to-date setup instructions!
 
+## Staging environment <Lifecycle status='beta' />
+
+:::note
+Currently in limited availability beta. Contact support or your account team if you're interested in beta access.
+:::
+
+Use a Staging environment to grant developers access to deployment workflows and tools while controlling access to production data. You can do this in a couple of ways, but the most straightforward is to configure Staging with a long-living branch (for example, `staging`) similar to but separate from the primary branch (for example, `main`). 
+
+In this scenario, the workflows would ideally move upstream from the Development environment -> Staging environment -> Production environment with developer branches feeding into the `staging` branch, then ultimately merging into `main`. In many cases, the `main` and `staging` branches will be identical after a merge and remain until the next batch of changes from the `development` branches are ready to be elevated. We recommend setting branch protection rules on `staging` similar to `main`.
+
+### Why use a staging environment
+
+There are two primary motivations for using a Staging environment:
+1. An additional validation layer before changes are deployed into Production. You can deploy, test, and explore your dbt models in Staging.
+2. Clear isolation between development workflows and production data. It enables developers to work in metadata-powered ways, using features like deferral and cross-project references, without accessing data in production deployments.
+
+:::info Coming soon: environment-level permissions
+Provide developers with the ability to create, edit, and trigger ad hoc jobs in the Staging environment, while keeping the Production environment locked down.
+:::
+
+Let's say you have `Project B` downstream of `Project A` with cross-project refs configured in the models. When developers work in the IDE for `Project B`, cross-project refs will resolve to the Staging environment of `Project A`, rather than production. You'll get the same results with those refs when jobs are run in the Staging environment. Only the Production environment will reference the Production data, keeping the data and access isolated without needing separate projects.
+
+If `Project B` also has a Staging deployment, then references to unbuilt upstream models within `Project B` will resolve to that environment, using [deferral](/docs/cloud/about-cloud-develop-defer), rather than resolving to the models in Production. This saves developers time and warehouse spend, while preserving clear separation of environments.
+
+Finally, the Staging environment has its own view in [dbt Explorer](/docs/collaborate/explore-projects), giving you a full view of your prod and pre-prod data.
+
+<Lightbox src="/img/docs/collaborate/dbt-explorer/explore-staging-env.png" width="85%" title="Explore in a staging environment" />
+
+
+### Create a Staging environment
+
+In the dbt Cloud, navigate to **Deploy** -> **Environments** and then click **Create Environment**. Select **Deployment** as the environment type. The option will be greyed out if you already have a development environment.
+
+<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/create-staging-environment.png" width="85%" title="Create a staging environment" />
+
+
+Follow the steps outlined in [deployment credentials](#deployment-connection) to complete the remainder of the environment setup.
+
+We recommend that the data warehouse credentials be for a dedicated user or service principal.
+
 
 ## Deployment connection
 
@@ -189,47 +229,6 @@ This section allows you to determine the credentials that should be used when co
 
 </WHCode>
 
-
-## Staging environment <Lifecycle status='beta' />
-
-:::note
-Currently in limited availability beta. Contact support or your account team if you're interested in beta access.
-:::
-
-Staging environments are useful ways to grant developers access to deployment workflows and tools while controlling access to production data. They are configured with their own long-living branch (for example, `staging`) that may be very similar to `main` in many ways while potentially limiting the data the developers can access. 
-
-Ideally, the workflows would move upstream from the Development environment -> Staging environment -> Production environment with developer branches feeding into the staging branch, then ultimately `main`. In many cases, the `main` and `staging` branches will be identical after a merge and remain until the next batch of changes from the `development` branches are ready to be elevated. We recommend setting branch protection rules on `staging` similar to `main`.
-
-### Create a staging environment 
-
-In the dbt Cloud, navigate to **Deploy** -> **Environments** and then click **Create Environment**. Select **Deployment** as the environment type. The option will be greyed out if you already have a development environment.
-
-<Lightbox src="/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/create-staging-environment.png" width="85%" title="Create a staging environment" />
-
-
-Follow the steps outlined in [deployment credentials](#deployment-connection) to complete the remainder of the environment setup.
-
-We recommend that the data warehouse credentials be for a dedicated user or service principal.
-
-### Why use a staging environment
-
-There are two primary motivations for using a Staging environment:
-1. An additional validation layer before changes are deployed into Production. You can deploy, test, and explore your dbt models in Staging.
-2. Clear isolation between development workflows and production data. It enables developers to work in metadata-powered ways, using features like deferral and cross-project references, without accessing data in production deployments.
-
-:::info Coming soon: environment-level permissions
-Provide developers with the ability to create, edit, and trigger ad hoc jobs in the Staging environment, while keeping the Production environment locked down.
-:::
-
-Let's say you have `Project B` downstream of `Project A` with cross-project refs configured in the models. When developers work in the IDE for `Project B`, cross-project refs will resolve to the Staging environment of `Project A`, rather than production. You'll get the same results with those refs when jobs are run in the Staging environment. Only the Production environment will reference the Production data, keeping the data and access isolated without needing separate projects.
-
-If `Project B` also has a Staging deployment, then references to unbuilt upstream models within `Project B` will resolve to that environment, using [deferral](/docs/cloud/about-cloud-develop-defer), rather than resolving to the models in Production. This saves developers time and warehouse spend, while preserving clear separation of environments.
-
-Finally, the Staging environment has its own view in [dbt Explorer](/docs/collaborate/explore-projects), giving you a full view of your prod and pre-prod data.
-
-<Lightbox src="/img/docs/collaborate/dbt-explorer/explore-staging-env.png" width="85%" title="Explore in a staging environment" />
-
-
 ## Related docs
 
 - [dbt Cloud environment best practices](/guides/set-up-ci)