From a104972e4328521bea164b59ffd0b2c5b97df0d9 Mon Sep 17 00:00:00 2001
From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com>
Date: Mon, 25 Mar 2024 09:13:41 +0000
Subject: [PATCH 001/131] update version block to display content
this PR fixes the version block so that the header AND content displays for v1.8 and higher. previously the header was displaying in the sidebar for all versions, but the actual content wasn't. this was a bit confusing and not the intended experience.
---
website/docs/docs/build/data-tests.md | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/website/docs/docs/build/data-tests.md b/website/docs/docs/build/data-tests.md
index 0261ea2d8b2..11a05f45def 100644
--- a/website/docs/docs/build/data-tests.md
+++ b/website/docs/docs/build/data-tests.md
@@ -125,7 +125,7 @@ You can find more information about these data tests, and additional configurati
Those four tests are enough to get you started. You'll quickly find you want to use a wider variety of tests—a good thing! You can also install generic data tests from a package, or write your own, to use (and reuse) across your dbt project. Check out the [guide on custom generic tests](/best-practices/writing-custom-generic-tests) for more information.
:::info
-There are generic tests defined in some open source packages, such as [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) and [dbt-expectations](https://hub.getdbt.com/calogica/dbt_expectations/latest/) — skip ahead to the docs on [packages](/docs/build/packages) to learn more!
+There are generic tests defined in some open-source packages, such as [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) and [dbt-expectations](https://hub.getdbt.com/calogica/dbt_expectations/latest/) — skip ahead to the docs on [packages](/docs/build/packages) to learn more!
:::
### Example
@@ -261,13 +261,13 @@ Note that, if you select to store test failures:
* Test result tables are created in a schema suffixed or named `dbt_test__audit`, by default. It is possible to change this value by setting a `schema` config. (For more details on schema naming, see [using custom schemas](/docs/build/custom-schemas).)
- A test's results will always **replace** previous failures for the same test.
-
+
## New `data_tests:` syntax
-Data tests were historically called "tests" in dbt as the only form of testing available. With the introduction of unit tests in v1.8, it was necessary to update our naming conventions and syntax. As of v1.8, `tests:` is still supported in your YML configuration file as an alias but will be deprecated in the future in favor of `data_tests:`.
+Data tests were historically called "tests" in dbt as the only form of testing available. With the introduction of unit tests in dbt version 1.8, it was necessary to update our naming conventions and syntax. As of dbt v1.8, `tests:` is still supported in your YML configuration file as an alias but will be deprecated in the future in favor of `data_tests:`.
-As we progress towards this deprecation, the examples in our docs pages will be updated to reflect this new syntax, but we highly recommend you begin the migration process as soon as you upgrade to v1.8 to avoid interruptions or issues in the future.
+As we progress towards this deprecation, we'll update the examples in our docs pages to reflect this new syntax, but we highly recommend you begin the migration process as soon as you upgrade to v1.8 to avoid interruptions or issues in the future.
```yml
@@ -281,6 +281,7 @@ models:
```
+
## FAQs
From dfb2e1d1e832537430b265655f5d426bac1d41d7 Mon Sep 17 00:00:00 2001
From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com>
Date: Mon, 25 Mar 2024 09:14:13 +0000
Subject: [PATCH 002/131] Update data-tests.md
---
website/docs/docs/build/data-tests.md | 2 --
1 file changed, 2 deletions(-)
diff --git a/website/docs/docs/build/data-tests.md b/website/docs/docs/build/data-tests.md
index 11a05f45def..6075e0d213a 100644
--- a/website/docs/docs/build/data-tests.md
+++ b/website/docs/docs/build/data-tests.md
@@ -278,8 +278,6 @@ models:
data_tests:
- unique
- not_null
-
-
```
From effd3fc1dd9ce6cb4a1b7f9f0dc55d58b6ad78a6 Mon Sep 17 00:00:00 2001
From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com>
Date: Mon, 25 Mar 2024 09:15:20 +0000
Subject: [PATCH 003/131] Update 01-upgrading-to-v1.8.md
fix link
---
.../docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md b/website/docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md
index e7004e893a0..d65f7a593a0 100644
--- a/website/docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md
+++ b/website/docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md
@@ -63,7 +63,7 @@ Unit tests are defined in YML files in your `models/` directory and are currentl
#### New `data_tests:` syntax
-The `tests:` syntax is changing to reflect the addition of unit tests. Start migrating your [data test](/docs/build/data-tests#new-test-syntax) YML to use `data_tests:` after you upgrade to v1.8 to prevent issues in the future.
+The `tests:` syntax is changing to reflect the addition of unit tests. Start migrating your [data test](/docs/build/data-tests#new-data_tests-syntax) YML to use `data_tests:` after you upgrade to v1.8 to prevent issues in the future.
```yml
From b5211dbbae6cd268aee0303650eb20291eb6670c Mon Sep 17 00:00:00 2001
From: mirnawong1
Date: Mon, 25 Mar 2024 12:43:11 +0000
Subject: [PATCH 004/131] fold in jerco's feedback
---
website/docs/guides/core-to-cloud-1.md | 47 ++++++++++++++++----------
1 file changed, 29 insertions(+), 18 deletions(-)
diff --git a/website/docs/guides/core-to-cloud-1.md b/website/docs/guides/core-to-cloud-1.md
index 212a44b0adb..49da689e4ac 100644
--- a/website/docs/guides/core-to-cloud-1.md
+++ b/website/docs/guides/core-to-cloud-1.md
@@ -12,7 +12,7 @@ recently_updated: true
---
## Introduction
-Moving from dbt Core to dbt Cloud streamlines analytics engineering workflows by allowing teams to develop, test, deploy, and explore data products using a single, fully managed software service.
+Moving from dbt Core to dbt Cloud streamlines analytics engineering workflows by allowing teams to develop, test, deploy, and explore data products using a single, fully managed platform.
dbt Cloud is the fastest and most reliable way to deploy dbt. It enables you to develop, test, deploy, and explore data products using a single, fully managed service. It also supports:
- The [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) or command line with [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) for development
@@ -22,7 +22,7 @@ dbt Cloud is the fastest and most reliable way to deploy dbt. It enables you to
Learn more about [dbt Cloud features](/docs/cloud/about-cloud/dbt-cloud-features).
-dbt Core is an open-source tool that enables data teams to transform data following analytics engineering best practices using a command line interface. It must be self-hosted and maintained.
+dbt Core is an open-source tool that enables data teams to transform data following analytics engineering best practices using a command line interface. dbt Core isn’t a managed platform offering, so it must be self-hosted and maintained. This requires significant, costly work that adds up over time to maintain and scale.
### What you'll learn
@@ -37,10 +37,12 @@ This guide outlines the steps you need to take to move from dbt Core to dbt Clou
- [Models configuration](https://docs.getdbt.com/guides/core-to-cloud-1?step=9#models-configuration): Get insights on validating and running your models in dbt Cloud, using either the dbt Cloud IDE or dbt Cloud CLI.
- [What's next?](https://docs.getdbt.com/guides/core-to-cloud-1?step=10#whats-next): Summarizes key takeaways and introduces what to expect in the following guides.
+If you are hosting dbt Core yourself, you need to answer these questions too. dbt Cloud provides a toolbox of options with sensible out-of-the-box defaults so you can focus on your data and analytics engineering workflows.
+
## Prerequisites
- You have an existing dbt Core project connected to a Git repository and data platform supported in [dbt Cloud](/docs/cloud/connect-data-platform/about-connections).
-- You are using dbt version 1.0 or later.
+- A supported version of dbt or select [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) of dbt.
- You have a dbt Cloud account. **[Don't have one? Start your free trial today](https://www.getdbt.com/signup)**!
### Related docs
@@ -61,9 +63,12 @@ This section outlines the steps to set up your dbt Cloud account and configure i
3. Configure [Single Sign-On (SSO)](/docs/cloud/manage-access/sso-overview) or [Role-based access control (RBAC)](/docs/cloud/manage-access/about-user-access#role-based-access-control) for easy and secure access.
- This removes the need to save passwords and secret environment variables locally.
-4. In **Account settings**, enable [partial parsing](/docs/deploy/deploy-environments#partial-parsing) to only reparse changed files, saving time.
+### Additional configuration
+Explore these additional configurations for performance and reliability improvements:
+
+1. In **Account settings**, enable [partial parsing](/docs/deploy/deploy-environments#partial-parsing) to only reparse changed files, saving time.
-5. In **Account settings**, enable [Git repo caching](/docs/deploy/deploy-environments#git-repository-caching) for job reliability & third-party outage protection.
+2. In **Account settings**, enable [Git repo caching](/docs/deploy/deploy-environments#git-repository-caching) for job reliability & third-party outage protection.
## Data platform setup
@@ -94,8 +99,10 @@ Your existing dbt project source code should live in a Git repository. In this s
1. Ensure your dbt project is in a Git repository.
-2. In **Account settings**, select **Integrations**, and [connect and configure Git](/docs/cloud/git/git-configuration-in-dbt-cloud) in dbt Cloud to connect your Git repository:
- - Connect with one of the [native integrations](/docs/cloud/git/git-configuration-in-dbt-cloud) in dbt Cloud (such as GitHub, GitLab, and Azure DevOps).
+2. In **Account settings**, select **Integrations** to [connect your Git repository](/docs/cloud/git/git-configuration-in-dbt-cloud) to dbt Cloud:
+ - (**Recommended**) Connect with one of the [native integrations](/docs/cloud/git/git-configuration-in-dbt-cloud) in dbt Cloud (such as GitHub, GitLab, and Azure DevOps).
+
+ This method is preferred for its simplicity, security features (including secure OAuth logins and automated workflows like CI builds on pull requests), and overall ease of use.
- [Import a Git repository](/docs/cloud/git/import-a-project-by-git-url) from any valid Git URL that points to a dbt project.
### Additional configuration
@@ -116,11 +123,14 @@ This section highlights the development configurations you’ll need for your db
### dbt Cloud environments
-The concept of an [environment](/docs/environments-in-dbt) in dbt Cloud is the same as a `target` in dbt Core.
-
-The primary difference between a dbt Cloud environment and a `target` in dbt Core is that you can make these configurations through the dbt Cloud UI instead of within the `profiles.yml` file.
+The most common data environments are production, staging, development. The way dbt Core manages [environments](/docs/environments-in-dbt) is through `target`, which are different sets of connection details.
-This difference streamlines the process of switching between development, staging, and production contexts, removing the need to manually edit the `profiles.yml` file. dbt Cloud environments also integrate with additional features such as job scheduling, version control, and more — making it easier to manage the full lifecycle of your dbt projects within a single platform. You can [set up](/reference/dbt-jinja-functions/target) or [customize](/docs/build/custom-target-names) target names in dbt Cloud.
+[dbt Cloud environments](/docs/dbt-cloud-environments) go further by:
+- Integrating with features such as job scheduling or version control — making it easier to manage the full lifecycle of your dbt projects within a single platform.
+- Streamlining the process of switching between development, staging, and production contexts.
+- Making it easy to configure environments through the dbt Cloud UI instead of manually editing `profiles.yml` file. You can also [set up](/reference/dbt-jinja-functions/target) or [customize](/docs/build/custom-target-names) target names in dbt Cloud.
+- Adding `profiles.yml` attributes to dbt Cloud environment settings with [Extended Attributes](/docs/dbt-cloud-environments#extended-attributes).
+- Using [Git repo caching](/docs/dbt-cloud-environments#git-repository-caching) to protect you from third-party outages, Git auth fails, and more.
### Initial setup steps
1. **Set up development environment** — Set up your [development](/docs/dbt-cloud-environments#create-a-development-environment) environment and [development credentials](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#access-the-cloud-ide). You’ll need this to access your dbt project and start developing.
@@ -137,7 +147,7 @@ Explore these additional configurations to optimize your developer setup further
1. **Custom target names** — If you’re using a [`custom target.name`](/reference/dbt-jinja-functions/target) in your project, we recommend you set them using [environment variables](/docs/build/environment-variables). Alternatively, you can update it at the developer credentials level.
### dbt Cloud commands
-1. Review the [dbt commands](/reference/dbt-commands) supported for dbt Cloud development. For example, `dbt debug` isn’t needed in dbt Cloud since the UI displays logs for your viewing.
+1. Review the [dbt commands](/reference/dbt-commands) supported for dbt Cloud development. For example, `dbt init` isn’t needed in dbt Cloud as you can create a new project directly in dbt Cloud.
## Environment variables
This section will help you understand how to set up and manage dbt Cloud environment variables for your project. The following categories are covered:
@@ -145,13 +155,14 @@ This section will help you understand how to set up and manage dbt Cloud environ
- [dbt Cloud environment variables order of precedence](/guides/core-to-cloud-1?step=7#dbt-cloud-environment-variables-order-of-precedence)
- [Set environment variables in dbt Cloud](/guides/core-to-cloud-1?step=7#set-environment-variables-in-dbt-cloud)
-In dbt Core, environment variables, or the [`env_var` function](/reference/dbt-jinja-functions/env_var), are defined in the `profiles.yml` file.
In dbt Cloud, you can set [environment variables](/docs/build/environment-variables) in the dbt Cloud user interface (UI). Read [Set up environment variables](#set-environment-variables-in-dbt-cloud) for more info.
+In dbt Core, environment variables, or the [`env_var` function](/reference/dbt-jinja-functions/env_var), are defined manually by the developer or within the external application running dbt.
+
### Environment variables in dbt Cloud
- dbt Cloud environment variables must be prefixed with `DBT_` (including `DBT_ENV_CUSTOM_ENV_` or `DBT_ENV_SECRET_`).
- If your dbt Core environment variables don’t follow this naming convention, perform a [“find and replace”](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#dbt-cloud-ide-features) in your project to make sure all references to these environment variables contain the proper naming conventions.
-- dbt Cloud secures environment variables, offering additional measures for sensitive values, such as prefixing keys with `DBT_ENV_SECRET_` to obscure them in logs and the UI.
+- dbt Cloud secures environment variables that enable more flexible configuration of data warehouse connections or git provider integrations, offering additional measures for sensitive values, such as prefixing keys with `DBT_ENV_SECRET_` to obscure them in logs and the UI.
@@ -186,7 +197,7 @@ To use the [dbt Cloud's job scheduler](/docs/deploy/job-scheduler), set up one e
1. **dbt Core version** — In your environment settings, configure dbt Cloud with the same dbt Core version.
- Once your full migration is complete, we recommend upgrading your environments to [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version-) to always get the latest features and more. You only need to do this once.
-2. **Configure your jobs** — [Create jobs](/docs/deploy/deploy-jobs#create-and-schedule-jobs) for automated or event-driven dbt jobs. You can use cron execution, manual, pull requests, or API triggers.
+2. **Configure your jobs** — [Create jobs](/docs/deploy/deploy-jobs#create-and-schedule-jobs) for scheduled or event-driven dbt jobs. You can use cron execution, manual, pull requests, or triggered on the completion of another job.
- Note that alongside [jobs in dbt Cloud](/docs/deploy/jobs), discover other ways to schedule and run your dbt jobs with the help of other tools. Refer to [Integrate with other tools](/docs/deploy/deployment-tools) for more information.
### Additional configuration
@@ -206,15 +217,15 @@ Explore these additional configurations to optimize your dbt Cloud orchestration
### CI/CD setup
-Building a custom solution to efficiently check code upon pull requests is complicated. With dbt Cloud, you can enable [continuous integration / continuous deployment (CI/CD)](/docs/deploy/continuous-integration) and configure dbt Cloud to run your dbt projects in a temporary schema when new commits are pushed to open pull requests.
+Building a custom solution to efficiently check code upon pull requests is complicated. With dbt Cloud, you can enable [continuous integration / continuous deployment (CI/CD)](/docs/deploy/continuous-integration) and configure dbt Cloud to run your dbt projects in a temporary schema when new commits are pushed to open pull requests.
This build-on-PR functionality is a great way to catch bugs before deploying to production, and an essential tool for data practitioners.
1. Set up an integration with a native Git application (such as Azure DevOps, GitHub, GitLab) and a CI environment in dbt Cloud.
-2. Create [a CI/CD job](/docs/deploy/ci-jobs) to optimize workflows.
+2. Create [a CI/CD job](/docs/deploy/ci-jobs) to automate quality checks before code is deployed to production.
3. Run your jobs in a production environment to fully implement CI/CD. Future pull requests will also leverage the last production runs to compare against.
-## Models configuration
+## Model development and discovery
In this section, you’ll be able to validate whether your models run or compile correctly in your development tool of choice: The [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) or [dbt Cloud CLI](/docs/cloud/cloud-cli-installation).
From 63a7a2d58410eadd4441dbeec6363bef7047be42 Mon Sep 17 00:00:00 2001
From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com>
Date: Thu, 4 Apr 2024 09:37:10 +0100
Subject: [PATCH 005/131] Update core-to-cloud-1.md
adding this as kolv has ga'ed
---
website/docs/guides/core-to-cloud-1.md | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/website/docs/guides/core-to-cloud-1.md b/website/docs/guides/core-to-cloud-1.md
index c36e1c89c2d..137049ae0e6 100644
--- a/website/docs/guides/core-to-cloud-1.md
+++ b/website/docs/guides/core-to-cloud-1.md
@@ -126,7 +126,7 @@ This section highlights the development configurations you’ll need for your db
### dbt Cloud environments
-The most common data environments are production, staging, development. The way dbt Core manages [environments](/docs/environments-in-dbt) is through `target`, which are different sets of connection details.
+The most common data environments are production, staging, and development. The way dbt Core manages [environments](/docs/environments-in-dbt) is through `target`, which are different sets of connection details.
[dbt Cloud environments](/docs/dbt-cloud-environments) go further by:
- Integrating with features such as job scheduling or version control — making it easier to manage the full lifecycle of your dbt projects within a single platform.
@@ -139,7 +139,7 @@ The most common data environments are production, staging, development. The way
1. **Set up development environment** — Set up your [development](/docs/dbt-cloud-environments#create-a-development-environment) environment and [development credentials](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#access-the-cloud-ide). You’ll need this to access your dbt project and start developing.
2. **dbt Core version** — In your dbt Cloud environment and credentials, use the same dbt Core version you use locally. You can run `dbt --version` in the command line to find out which version of dbt Core you’re using.
- - Once your full migration is complete, consider upgrading your environments to [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version-) to always get the latest features and more.
+ - When using dbt Core, you need to think about which version you’re using, and manage your own upgrades. When using dbt Cloud and [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version-), you don’t have to.
3. **Connect to your data platform** — When using dbt Cloud, you can [connect to your data platform](/docs/cloud/connect-data-platform/about-connections) directly in the UI.
- Each environment is roughly equivalent to an entry in your `profiles.yml` file. This means you don't need a `profiles.yml` file in your project.
@@ -268,4 +268,4 @@ For next steps, we'll soon share other guides on how to manage your move and tip
-
\ No newline at end of file
+
From 742160896b06bd15e8eb8190b908046d9e0af28d Mon Sep 17 00:00:00 2001
From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com>
Date: Thu, 4 Apr 2024 11:43:44 +0100
Subject: [PATCH 006/131] Update website/docs/guides/core-to-cloud-1.md
---
website/docs/guides/core-to-cloud-1.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/guides/core-to-cloud-1.md b/website/docs/guides/core-to-cloud-1.md
index 137049ae0e6..45100c2579f 100644
--- a/website/docs/guides/core-to-cloud-1.md
+++ b/website/docs/guides/core-to-cloud-1.md
@@ -139,7 +139,7 @@ The most common data environments are production, staging, and development. The
1. **Set up development environment** — Set up your [development](/docs/dbt-cloud-environments#create-a-development-environment) environment and [development credentials](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#access-the-cloud-ide). You’ll need this to access your dbt project and start developing.
2. **dbt Core version** — In your dbt Cloud environment and credentials, use the same dbt Core version you use locally. You can run `dbt --version` in the command line to find out which version of dbt Core you’re using.
- - When using dbt Core, you need to think about which version you’re using, and manage your own upgrades. When using dbt Cloud and [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version-), you don’t have to.
+ - When using dbt Core, you need to think about which version you’re using, and manage your own upgrades. When using dbt Cloud, leverage [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version-) so you don’t have to.
3. **Connect to your data platform** — When using dbt Cloud, you can [connect to your data platform](/docs/cloud/connect-data-platform/about-connections) directly in the UI.
- Each environment is roughly equivalent to an entry in your `profiles.yml` file. This means you don't need a `profiles.yml` file in your project.
From ccbb5394180261753cb766c2d732f71dfcdb273d Mon Sep 17 00:00:00 2001
From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com>
Date: Thu, 4 Apr 2024 11:46:45 +0100
Subject: [PATCH 007/131] Update core-to-cloud-1.md
---
website/docs/guides/core-to-cloud-1.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/website/docs/guides/core-to-cloud-1.md b/website/docs/guides/core-to-cloud-1.md
index 45100c2579f..93ccde660e4 100644
--- a/website/docs/guides/core-to-cloud-1.md
+++ b/website/docs/guides/core-to-cloud-1.md
@@ -139,7 +139,7 @@ The most common data environments are production, staging, and development. The
1. **Set up development environment** — Set up your [development](/docs/dbt-cloud-environments#create-a-development-environment) environment and [development credentials](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#access-the-cloud-ide). You’ll need this to access your dbt project and start developing.
2. **dbt Core version** — In your dbt Cloud environment and credentials, use the same dbt Core version you use locally. You can run `dbt --version` in the command line to find out which version of dbt Core you’re using.
- - When using dbt Core, you need to think about which version you’re using, and manage your own upgrades. When using dbt Cloud, leverage [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version-) so you don’t have to.
+ - When using dbt Core, you need to think about which version you’re using and manage your own upgrades. When using dbt Cloud, leverage [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version-) so you don’t have to.
3. **Connect to your data platform** — When using dbt Cloud, you can [connect to your data platform](/docs/cloud/connect-data-platform/about-connections) directly in the UI.
- Each environment is roughly equivalent to an entry in your `profiles.yml` file. This means you don't need a `profiles.yml` file in your project.
@@ -147,7 +147,7 @@ The most common data environments are production, staging, and development. The
### Additional configuration
Explore these additional configurations to optimize your developer setup further:
-1. **Custom target names** — If you’re using a [`custom target.name`](/reference/dbt-jinja-functions/target) in your project, we recommend you set them using [environment variables](/docs/build/environment-variables). Alternatively, you can update it at the developer credentials level.
+1. **Custom target names** — If you’re using a [`custom target.name`](/reference/dbt-jinja-functions/target) in your project, we recommend setting them at the developer credentials level. Alternatively, you can set them using [environment variables](/docs/build/environment-variables).
### dbt Cloud commands
1. Review the [dbt commands](/reference/dbt-commands) supported for dbt Cloud development. For example, `dbt init` isn’t needed in dbt Cloud as you can create a new project directly in dbt Cloud.
From d9140245ca8e1ff2040749648bc7cc79d40cf8f8 Mon Sep 17 00:00:00 2001
From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com>
Date: Thu, 4 Apr 2024 11:58:14 +0100
Subject: [PATCH 008/131] Update website/docs/guides/core-to-cloud-1.md
---
website/docs/guides/core-to-cloud-1.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/guides/core-to-cloud-1.md b/website/docs/guides/core-to-cloud-1.md
index 93ccde660e4..adef7d50912 100644
--- a/website/docs/guides/core-to-cloud-1.md
+++ b/website/docs/guides/core-to-cloud-1.md
@@ -45,7 +45,7 @@ If you are hosting dbt Core yourself, you need to answer these questions too. db
## Prerequisites
- You have an existing dbt Core project connected to a Git repository and data platform supported in [dbt Cloud](/docs/cloud/connect-data-platform/about-connections).
-- A supported version of dbt or select [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) of dbt.
+- A supported version of dbt or select [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) of dbt.
- You have a dbt Cloud account. **[Don't have one? Start your free trial today](https://www.getdbt.com/signup)**!
### Related docs
From eefc11c274df445b7bf41916790ff565f57fc225 Mon Sep 17 00:00:00 2001
From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com>
Date: Thu, 4 Apr 2024 12:06:42 +0100
Subject: [PATCH 009/131] Update core-to-cloud-1.md
---
website/docs/guides/core-to-cloud-1.md | 1 +
1 file changed, 1 insertion(+)
diff --git a/website/docs/guides/core-to-cloud-1.md b/website/docs/guides/core-to-cloud-1.md
index adef7d50912..c8b75b56287 100644
--- a/website/docs/guides/core-to-cloud-1.md
+++ b/website/docs/guides/core-to-cloud-1.md
@@ -7,6 +7,7 @@ icon: 'guides'
time_to_complete: 'Total estimated time: 3-4 hours'
hide_table_of_contents: true
tags: ['Migration','dbt Core','dbt Cloud']
+keywords: ['dbt Core','dbt Cloud','Migration', 'Move dbt', 'Migrate dbt']
level: 'Intermediate'
recently_updated: true
---
From 21416245753071aca4b0c2ffe985eee1d433c4db Mon Sep 17 00:00:00 2001
From: mirnawong1
Date: Thu, 4 Apr 2024 12:56:10 +0100
Subject: [PATCH 010/131] add part 2
---
website/docs/guides/core-to-cloud-1.md | 6 +-
website/docs/guides/core-to-cloud-2.md | 167 +++++++++++++++++++
website/snippets/_cloud-environments-info.md | 2 +-
3 files changed, 170 insertions(+), 5 deletions(-)
create mode 100644 website/docs/guides/core-to-cloud-2.md
diff --git a/website/docs/guides/core-to-cloud-1.md b/website/docs/guides/core-to-cloud-1.md
index f2ee7b016e2..0028d4c61fb 100644
--- a/website/docs/guides/core-to-cloud-1.md
+++ b/website/docs/guides/core-to-cloud-1.md
@@ -241,7 +241,7 @@ You have learned:
- How to configure your development, orchestration, and CI/CD environments
- How to set up environment variables and validate your models.
-For next steps, we'll soon share other guides on how to manage your move and tips/faqs. Stay tuned!
+Review the next guide, [Move from dbt Core to dbt Cloud: Everything you need to know](/docs/guides/core-to-cloud-2), in this series to learn about the strategic considerations you need to take to move from dbt Core to dbt Cloud.
+
### Related docs
- [Learn dbt Cloud](https://courses.getdbt.com/collections)
From 865fb29635686b0f0f360a605e6b8b9ba945feca Mon Sep 17 00:00:00 2001
From: mirnawong1
Date: Fri, 5 Apr 2024 11:01:46 +0100
Subject: [PATCH 026/131] updates to guide 2
---
website/docs/guides/core-to-cloud-2.md | 88 ++++++++++---------
website/docs/guides/core-to-cloud-3.md | 12 +--
.../snippets/_core-to-cloud-guide-table.md | 5 ++
3 files changed, 55 insertions(+), 50 deletions(-)
create mode 100644 website/snippets/_core-to-cloud-guide-table.md
diff --git a/website/docs/guides/core-to-cloud-2.md b/website/docs/guides/core-to-cloud-2.md
index e3148c2b760..ff186b0b4ca 100644
--- a/website/docs/guides/core-to-cloud-2.md
+++ b/website/docs/guides/core-to-cloud-2.md
@@ -16,29 +16,29 @@ Moving from dbt Core to dbt Cloud streamlines analytics engineering workflows by
Explore our 3-guide series on moving from dbt Core to dbt Cloud. The series is ideal for users aiming for streamlined workflows and enhanced analytics:
-| Guide | Information | Audience |
-|------------|-------------|----------|
-| [Move from dbt Core to dbt Cloud: Get started](/guides/core-to-cloud-1?step=1) | Learn the steps needed to move from dbt Core to dbt Cloud. | Developers, Data engineers, Data analysts |
-| [Move from dbt Core to dbt Cloud: What you need to know](/docs/guides/core-to-cloud-2) | Understand the considerations and methods needed. | Team leads, Admins |
-| [Move from dbt Core to dbt Cloud: Optimization tips] | Learn how to optimize your dbt Cloud experience with common scenarios and useful tips. | Everyone |
+import CoretoCloudTable from '/snippets/_core-to-cloud-guide-table.md';
-### What you'll learn
+
+
+## What you'll learn
Migrating from dbt Core to dbt Cloud can yield significant benefits, and a standard way of working. Understanding the differences between dbt Cloud and your current Core deployment will help you strategize and plan for your move.
The guide outlines the following steps:
-- [Considerations](https://docs.getdbt.com/guides/core-to-cloud-1?step=3#account-setup): Learn about the most important things you need to think about when moving from Core to Cloud.
-- [Plan your move](https://docs.getdbt.com/guides/core-to-cloud-1?step=4#data-platform-setup): Considerations you need to make, such as user roles and permissions, onboarding order, current workflows, and more.
-- [Move to dbt Cloud](https://docs.getdbt.com/guides/core-to-cloud-1?step=5#git-setup): Review the steps to move your dbt Core project to dbt Cloud, including setting up your account, data platform, and Git repository.
-- [Test and validate](https://docs.getdbt.com/guides/core-to-cloud-1?step=6#developer-setup): Discover how to ensure model accuracy and performance post-move,.
-- [Transition and training](https://docs.getdbt.com/guides/core-to-cloud-1?step=8#orchestration-setup): Learn how to fully transition to dbt Cloud and what training and support you may need.
-- [Beyond your move](https://docs.getdbt.com/guides/core-to-cloud-1?step=9#models-configuration): Get insights on additional features you can use to unlock the full potential of dbt Cloud.
-- [Summary](link): Summarizes key takeaways and what you've learned in this guide.
-- [What's next?](https://docs.getdbt.com/guides/core-to-cloud-1?step=10#whats-next): Introduces what to expect in the following guides.
+- [Considerations](https://docs.getdbt.com/guides/core-to-cloud-2?step=3): Learn about the most important things you need to think about when moving from Core to Cloud.
+- [Plan your move](https://docs.getdbt.com/guides/core-to-cloud-2?step=4): Considerations you need to make, such as user roles and permissions, onboarding order, current workflows, and more.
+- [Move to dbt Cloud](https://docs.getdbt.com/guides/core-to-cloud-2?step=5): Review the steps to move your dbt Core project to dbt Cloud, including setting up your account, data platform, and Git repository.
+- [Test and validate](https://docs.getdbt.com/guides/core-to-cloud-2?step=6): Discover how to ensure model accuracy and performance post-move,.
+- [Transition and training](https://docs.getdbt.com/guides/core-to-cloud-2?step=7): Learn how to fully transition to dbt Cloud and what training and support you may need.
+- [Summary](https://docs.getdbt.com//guides/core-to-cloud-2?step=8): Summarizes key takeaways and what you've learned in this guide.
+- [What's next?](https://docs.getdbt.com//guides/core-to-cloud-2?step=9): Introduces what to expect in the following guides.
## Considerations
-If your team has a custom deployment of dbt Core today, you could be reading this guide because you’ve realized the burden of maintaining that deployment, because the person who set it up has since left, or because you’re interested in what dbt Cloud could do to standardize your deployment and consolidate some of the toolings that you may already have in place.
+If your team has a custom deployment of dbt Core today, you could be reading this guide because:
+- You’ve realized the burden of maintaining that deployment.
+- The person who set it up has since left.
+- Or you’re interested in what dbt Cloud could do to standardize your deployment and consolidate some of the toolings that you may already have in place.
This guide shares the technical adjustments and team collaboration strategies you’ll need to know to move your project from dbt Core to dbt Cloud. Each “BYO” (build your own) deployment of dbt Core will look a little different, but after seeing hundreds of teams make the migration, there are many things in common.
@@ -52,7 +52,13 @@ The most important things you need to think about when moving from dbt Core to d
- How do your data developers prefer to work?
- How do you manage different data environments, and different behaviors in those environments?
-dbt Cloud provides standard mechanisms for tackling the considerations above: cross-team collaboration, access control, orchestration, isolated data environments. which delivers long-term benefits to your organization. If you have rolled your own dbt Core deployment, you have probably come up with different answers.
+dbt Cloud provides standard mechanisms for tackling these considerations, all of which delivers long-term benefits to your organization:
+- cross-team collaboration
+- access control
+- orchestration
+- isolated data environments
+
+If you have rolled your own dbt Core deployment, you have probably come up with different answers.
## Plan your move
@@ -114,9 +120,11 @@ This provides developers with the benefits of an enhanced workflow while ensurin
## Move to dbt Cloud
-After planning your move and reviewing the considerations, you may want to start moving your dbt Core project to dbt Cloud — check out the detailed [Move to dbt Cloud: Get started](/guides/core-to-cloud-1?step=1) guide for useful tasks and insights for a smooth move from dbt Core to dbt Cloud.
+After reviewing the considerations and planning your move, you may want to start moving your dbt Core project to dbt Cloud — check out the detailed [Move to dbt Cloud: Get started](/guides/core-to-cloud-1?step=1) guide for useful tasks and insights for a smooth move from dbt Core to dbt Cloud.
+
+This guide is your roadmap to help you think about migration strategies and what moving from dbt Core to dbt Cloud would look like.
-This guide is your roadmap to help you think about migration strategies and what moving from dbt Core to dbt Cloud would look like. For a more detailed comparison of dbt Core and dbt Cloud, check out [How dbt Cloud compares with dbt Core](https://www.getdbt.com/product/dbt-core-vs-dbt-cloud).
+For a more detailed comparison of dbt Core and dbt Cloud, check out [How dbt Cloud compares with dbt Core](https://www.getdbt.com/product/dbt-core-vs-dbt-cloud).
## Test and validate
@@ -125,38 +133,35 @@ After setting the foundations of dbt Cloud, it's important to validate your migr
- **Review your dbt project:** Ensure your project compiles correctly and that you can run commands. Make sure your models are accurate and monitor performance post-move.
- **Start cutover:** You can start the cutover to dbt Cloud by creating a dbt Cloud job with commands that only run a small subset of the DAG. Validate the tables are being populated in the proper database/schemas as expected. Then continue to expand the scope of the job to include more sections of the DAG as you gain confidence in the results.
- **Precision in testing:** Use [unit testing](/docs/build/unit-tests) to allow you to validate your SQL modeling logic on a small set of static inputs *before* you materialize your full model in production.
-- **Access and permissions**: Review and adjust access controls and permissions within dbt Cloud to maintain security protocols and safeguard your data.
+- **Access and permissions**: Review and adjust [access controls and permissions](/docs/cloud/manage-access/about-user-access) within dbt Cloud to maintain security protocols and safeguard your data.
## Transition and training
-Once you’ve confirmed that dbt Cloud orchestration and CI/CD are working as expected, you should pause your current orchestration tool and stop or update your current CI/CD process. This is not relevant if you’re still using an external orchestrator (such as Airflow), and you’ve swapped out dbt-core execution for dbt Cloud execution (through the [API](/docs/dbt-cloud-apis/overview)).
+Once you’ve confirmed that dbt Cloud orchestration and CI/CD are working as expected, you should pause your current orchestration tool and stop or update your current CI/CD process. This is not relevant if you’re still using an external orchestrator (such as Airflow), and you’ve swapped out `dbt-core` execution for dbt Cloud execution (through the [API](/docs/dbt-cloud-apis/overview)).
Familiarize your team with dbt Cloud's [features](/docs/cloud/about-cloud/dbt-cloud-features) and optimize development and deployment processes. Some key features to consider include:
- **Version management:** Manage [dbt versions](/docs/dbt-versions/upgrade-dbt-version-in-cloud) and ensure team synchronization with dbt Cloud's one-click feature, eliminating the hassle of manual updates and version discrepancies. You can also **[Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version)** to always get the latest fixes and early access to new functionality for your dbt project.
-- **Documentation and Source freshness:** Benefit from automatic storage of documentation and source freshness metrics in dbt Cloud, eliminating the need for manual configuration.
- - Documentation is stored without any configuration needed with [dbt Explorer](/docs/collaborate/explore-projects) (once you have a successful job in a Production environment).
- - [Source freshness](/docs/deploy/source-freshness) is stored without any configuration needed.
-- **Notifications and logs:** Receive immediate [notifications](/docs/deploy/monitor-jobs) for job failures, with direct links to job details. Access comprehensive logs for all job runs to facilitate troubleshooting.
+- **Development tools**: Use the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) or [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) to build, test, run, and version control your dbt projects.
+- **Documentation and Source freshness:** Automate storage of documentation and track [source freshness](/docs/deploy/source-freshness) in dbt Cloud, streamlining project maintenance.
+- **Notifications and logs:** Receive immediate [notifications](/docs/deploy/monitor-jobs) for job failures, with direct links to job details. Access comprehensive logs for all job runs to help with troubleshooting.
+- **CI/CD:** Use dbt Cloud's [CI/CD](/docs/deploy/ci-jobs) feature to run your dbt projects in a temporary schema when new commits are pushed to open pull requests, helpful in catching bugs before deploying to production.
-### Advanced features and API use
+### Beyond your move
+
+Now that you’ve chosen dbt Cloud as your platform, you’ve unlocked the power of streamlining collaboration, enhancing workflow efficiency, and leveraging powerful [features](/docs/cloud/about-cloud/dbt-cloud-features) for analytics engineering teams. Here are some additional features you can use to unlock the full potential of dbt Cloud:
- **Audit logs:** Use [audit logs](/docs/cloud/manage-access/audit-log) to review actions performed by people in your organization. Audit logs contain audited user and system events in real time. You can even export *all* the activity (beyond the 90 days you can view in dbt Cloud).
-- **dbt Cloud APIs:** Use dbt Cloud's robust [APIs](/docs/dbt-cloud-apis/overview) to Create, Read, Update, and Delete (CRUD) projects/jobs/environments project. The [dbt Cloud Administrative API](/docs/dbt-cloud-apis/admin-cloud-api) and [Terraform provider](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest/docs/resources/job) facilitate programmatic access and configuration storage, while the [Discovery API](/docs/dbt-cloud-apis/discovery-api) offers extensive metadata querying capabilities, such as job data, model configurations, usage, and overall project health.
-- **dbt Semantic Layer:** The [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) allows you to define universal metrics on top of your models, that can subsequently be queried in your BI layer. This means no more inconsistent metrics — there’s now a centralized way to define these metrics and create visibility in every component of the data flow.
+- **dbt Cloud APIs:** Use dbt Cloud's robust [APIs](/docs/dbt-cloud-apis/overview) to Create, Read, Update, and Delete (CRUD) projects/jobs/environments project. The [dbt Cloud Administrative API](/docs/dbt-cloud-apis/admin-cloud-api) and [Terraform provider](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest/docs/resources/job) facilitate programmatic access and configuration storage. While the [Discovery API](/docs/dbt-cloud-apis/discovery-api) offers extensive metadata querying capabilities, such as job data, model configurations, usage, and overall project health.
+- **dbt Explorer**: Use [dbt Explorer](/docs/collaborate/explore-projects) to view your project's [resources](/docs/build/projects) (such as models, tests, and metrics) and their [lineage](https://docs.getdbt.com/terms/data-lineage) to gain a better understanding of its latest production state. (Once you have a successful job in a Production environment).
+- **dbt Semantic Layer:** The [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) allows you to define universal metrics on top of your models, that can then be queried in your business intelligent (BI) tool. This means no more inconsistent metrics — there’s now a centralized way to define these metrics and create visibility in every component of the data flow.
+- **dbt Mesh:** Use [dbt Mesh](/best-practices/how-we-mesh/mesh-1-intro) to share data models across organizations, enabling data teams to collaborate on shared data models and leverage the work of other teams.
### Additional help
+- **dbt Learn courses**: Access our free [Learn dbt Cloud](https://courses.getdbt.com/collections) video courses for on-demand training.
+- **dbt Community:** Join the [dbt Community](https://community.getdbt.com/) to connect with other dbt users, ask questions, and share best practices.
- **dbt Support team:** Our dbt Support team is always available to help you troubleshoot your dbt Cloud issues. Create a support ticket in dbt Cloud and we’ll be happy to help!
-- **Solutions Architect:** You have Solutions Architect available to help you troubleshoot solutions and account management help for you and your team.
-
-## Beyond your move
-
-Now that you’ve chosen dbt Cloud as your platform, you’ve unlocked the power of streamlining collaboration, enhancing workflow efficiency, and leveraging powerful [features](/docs/cloud/about-cloud/dbt-cloud-features) for analytics engineering teams. Here are some additional features you can use to unlock the full potential of dbt Cloud:
-
-- Use dbt Cloud’s [advanced monitoring](/docs/deploy/monitor-jobs) to receive immediate notifications for job failures, with direct links to job details. Access comprehensive logs for all job runs to facilitate troubleshooting.
-- Implement the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) for consistent, centralized metric definitions.
-- Use [dbt Explorer](/docs/collaborate/explore-projects) to view your project's [resources](/docs/build/projects) (such as models, tests, and metrics) and their [lineage](https://docs.getdbt.com/terms/data-lineage) to gain a better understanding of its latest production state. Lifecycle status="team,enterprise"/>
-- Use dbt Cloud's robust [APIs](/docs/dbt-cloud-apis/overview) to Create, Read, Update, and Delete (CRUD) projects/jobs/environments project. The [dbt Cloud Administrative API](/docs/dbt-cloud-apis/admin-cloud-api) and [Terraform provider](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest/docs/resources/job) facilitate programmatic access and configuration storage, while the [Discovery API](/docs/dbt-cloud-apis/discovery-api) offers extensive metadata querying capabilities, such as job data, model configurations, usage, and overall project health.
+- **Account management** Enterprise accounts have an account management team available to help troubleshoot solutions and account management assistance.
## Summary
@@ -166,11 +171,11 @@ This guide should now have given you some insight and equipped you with a framew
- **Plan you move**: Highlighting the importance of workflow redesign, role-specific responsibilities, and the adoption of new processes to harness dbt Cloud's collaborative and efficient environment.
-- **Move to dbt Cloud**: Outlining the technical steps required to transition your dbt Core project to dbt Cloud, including setting up your account, data platform, and Git repository.
+- **Move to dbt Cloud**: Linking to the guide that outlines technical steps required to transition your dbt Core project to dbt Cloud, including setting up your account, data platform, and Git repository.
-- **Test and validate**: Emphasizing technical transitions, including the essential task of testing and validating your dbt projects within the dbt Cloud ecosystem to ensure data integrity and performance.
+- **Test and validate**: Emphasizing technical transitions, including testing and validating your dbt projects within the dbt Cloud ecosystem to ensure data integrity and performance.
-- **Training**: Share useful transition, training, and onboarding information for your team. Fully leverage dbt Cloud's capabilities, from development tools (dbt Cloud CLI and dbt Cloud IDE) to advanced features such as dbt Explorer, the Semantic Layer, and dbt Mesh.
+- **Transition and training**: Share useful transition, training, and onboarding information for your team. Fully leverage dbt Cloud's capabilities, from development tools (dbt Cloud CLI and dbt Cloud IDE) to advanced features such as dbt Explorer, the Semantic Layer, and dbt Mesh.
## What’s next?
@@ -180,8 +185,7 @@ Congratulations on finishing this guide, we hope it's given you insight into the
For next steps, you can continue exploring our 3-part-guide series on moving from dbt Core to dbt Cloud:
-- Begin with [Get started](/guides/core-to-cloud-1?step=11) to learn how to move from dbt Core to dbt Cloud and what you need to get started.
-- Conclude with Optimization tips to learn how to optimize your dbt Cloud experience and get answers to common questions.
+
### Related docs
- [Learn dbt Cloud](https://courses.getdbt.com/collections) courses
diff --git a/website/docs/guides/core-to-cloud-3.md b/website/docs/guides/core-to-cloud-3.md
index 5f35069f85d..731f603ba3f 100644
--- a/website/docs/guides/core-to-cloud-3.md
+++ b/website/docs/guides/core-to-cloud-3.md
@@ -16,11 +16,9 @@ Moving from dbt Core to dbt Cloud streamlines analytics engineering workflows by
Explore our 3-guide series on moving from dbt Core to dbt Cloud. The series is ideal for users aiming for streamlined workflows and enhanced analytics:
-| Guide | Information | Audience |
-|------------|-------------|----------|
-| [Move from dbt Core to dbt Cloud: Get started](/guides/core-to-cloud-1?step=1) | Learn the steps needed to move from dbt Core to dbt Cloud. | Developers, Data engineers, Data analysts |
-| [Move from dbt Core to dbt Cloud: What you need to know](/docs/guides/core-to-cloud-2) | Understand the considerations and methods needed. | Team leads, Admins |
-| [Move from dbt Core to dbt Cloud: Optimization tips] | Learn how to optimize your dbt Cloud experience with common scenarios and useful tips. | Everyone |
+import CoretoCloudTable from '/snippets/_core-to-cloud-guide-table.md';
+
+
This guide shares some tips, enhancements, technical adjustments, and frequently asked questions that you may encounter during your move.
@@ -113,9 +111,7 @@ Congratulations on making it through the Move from dbt Core to dbt Cloud guide
For next steps, you can continue exploring our 3-part-guide series on moving from dbt Core to dbt Cloud:
-- Begin with [Get started](/guides/core-to-cloud-1?step=11) to learn how to move from dbt Core to dbt Cloud and what you need to get started.
-- Move on to [Everything you need to know](/docs/guides/core-to-cloud-2) to understand the considerations and methods you need to move from dbt Core to dbt Cloud.
-- Conclude with Optimization tips to learn how to optimize your dbt Cloud experience and get answers to common questions.
+
diff --git a/website/snippets/_core-to-cloud-guide-table.md b/website/snippets/_core-to-cloud-guide-table.md
new file mode 100644
index 00000000000..4775218805c
--- /dev/null
+++ b/website/snippets/_core-to-cloud-guide-table.md
@@ -0,0 +1,5 @@
+| Guide | Information | Audience |
+|------------|-------------|----------|
+| [Move from dbt Core to dbt Cloud: Get started](/guides/core-to-cloud-1?step=1) | Learn the steps needed to move from dbt Core to dbt Cloud. | Developers, Data engineers, Data analysts |
+| [Move from dbt Core to dbt Cloud: What you need to know](/docs/guides/core-to-cloud-2) | Understand the considerations and methods needed in your move. | Team leads, Admins |
+| [Move from dbt Core to dbt Cloud: Optimization tips](/guides/core-to-cloud-3) | Learn how to optimize your dbt Cloud experience with common scenarios and useful tips. | Everyone |
From 7464addf590106f05fe074655e09e5f0ea4c580d Mon Sep 17 00:00:00 2001
From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com>
Date: Fri, 5 Apr 2024 11:40:21 +0100
Subject: [PATCH 027/131] Update core-to-cloud-1.md
---
website/docs/guides/core-to-cloud-1.md | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/website/docs/guides/core-to-cloud-1.md b/website/docs/guides/core-to-cloud-1.md
index de97282bb06..45057ef5937 100644
--- a/website/docs/guides/core-to-cloud-1.md
+++ b/website/docs/guides/core-to-cloud-1.md
@@ -16,7 +16,7 @@ recently_updated: true
## Introduction
-Explore our 3-guide series on moving from dbt Core to dbt Cloud. The series is ideal for users aiming for streamlined workflows and enhanced analytics:
+Explore our 3-part-guide series on moving from dbt Core to dbt Cloud. The series is ideal for users aiming for streamlined workflows and enhanced analytics:
import CoretoCloudTable from '/snippets/_core-to-cloud-guide-table.md';
@@ -259,15 +259,15 @@ You have learned:
- How to set up your dbt Cloud account
- How to connect your data platform and Git repository
- How to configure your development, orchestration, and CI/CD environments
-- How to set up environment variables and validate your models.
+- How to set up environment variables and validate your models
For next steps, you can continue exploring our 3-part-guide series on moving from dbt Core to dbt Cloud:
### Related docs
-- [Learn dbt Cloud](https://courses.getdbt.com/collections)
-- Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights
+- [Learn dbt Cloud](https://courses.getdbt.com/collections) video courses for on-demand learning.
+- Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights.
- Work with the [dbt Labs’ Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and migration.
From 0b6b44a336864e3e3d8b037c2eabe1293ded98a0 Mon Sep 17 00:00:00 2001
From: mirnawong1
Date: Fri, 5 Apr 2024 11:53:24 +0100
Subject: [PATCH 028/131] updates
---
website/docs/guides/core-to-cloud-2.md | 6 +-
website/docs/guides/core-to-cloud-3.md | 164 +++++++++++++++----------
2 files changed, 101 insertions(+), 69 deletions(-)
diff --git a/website/docs/guides/core-to-cloud-2.md b/website/docs/guides/core-to-cloud-2.md
index ff186b0b4ca..214352410bc 100644
--- a/website/docs/guides/core-to-cloud-2.md
+++ b/website/docs/guides/core-to-cloud-2.md
@@ -14,7 +14,7 @@ recently_updated: true
## Introduction
Moving from dbt Core to dbt Cloud streamlines analytics engineering workflows by allowing teams to develop, test, deploy, and explore data products using a single, fully managed platform.
-Explore our 3-guide series on moving from dbt Core to dbt Cloud. The series is ideal for users aiming for streamlined workflows and enhanced analytics:
+Explore our 3-part-guide series on moving from dbt Core to dbt Cloud. The series is ideal for users aiming for streamlined workflows and enhanced analytics:
import CoretoCloudTable from '/snippets/_core-to-cloud-guide-table.md';
@@ -92,7 +92,7 @@ If you have multiple teams of dbt developers, think about how to start your onbo
Discover how dbt Cloud can help simplify development, orchestration, and testing.
- **Development**: Develop dbt models, allowing you to build, test, run, and version control your dbt projects using the dbt Cloud CLI (command line interface or code editor) or dbt Cloud IDE (browser-based).
-- **Orchestration**: Create custom schedules to run your production jobs. Schedule jobs by day of the week, time of day, or a recurring interval.
+- **Orchestration**: Create custom schedules to run your production jobs. Schedule jobs by day of the week, time of day, or a recurring interval.
- Set up CI to ensure developer effectiveness, and CD jobs to deploy changes as soon as they’re merged.
- Link deploy jobs together by [triggering a job](/docs/deploy/deploy-jobs#trigger-on-job-completion) when another one is completed.
- For the most flexibility, you can use the [dbt Cloud API](https://docs.getdbt.com/dbt-cloud/api-v2#/) to trigger jobs. This makes sense when you want to integrate dbt execution with other data workflows.
@@ -161,7 +161,7 @@ Now that you’ve chosen dbt Cloud as your platform, you’ve unlocked the power
- **dbt Learn courses**: Access our free [Learn dbt Cloud](https://courses.getdbt.com/collections) video courses for on-demand training.
- **dbt Community:** Join the [dbt Community](https://community.getdbt.com/) to connect with other dbt users, ask questions, and share best practices.
- **dbt Support team:** Our dbt Support team is always available to help you troubleshoot your dbt Cloud issues. Create a support ticket in dbt Cloud and we’ll be happy to help!
-- **Account management** Enterprise accounts have an account management team available to help troubleshoot solutions and account management assistance.
+- **Account management** Enterprise accounts have an account management team available to help troubleshoot solutions and account management assistance. [Book a demo](https://www.getdbt.com/contact) to learn more.
## Summary
diff --git a/website/docs/guides/core-to-cloud-3.md b/website/docs/guides/core-to-cloud-3.md
index 731f603ba3f..084a66a3708 100644
--- a/website/docs/guides/core-to-cloud-3.md
+++ b/website/docs/guides/core-to-cloud-3.md
@@ -12,102 +12,134 @@ recently_updated: true
---
## Introduction
+
Moving from dbt Core to dbt Cloud streamlines analytics engineering workflows by allowing teams to develop, test, deploy, and explore data products using a single, fully managed platform.
-Explore our 3-guide series on moving from dbt Core to dbt Cloud. The series is ideal for users aiming for streamlined workflows and enhanced analytics:
+Explore our 3-part-guide series on moving from dbt Core to dbt Cloud. The series is ideal for users aiming for streamlined workflows and enhanced analytics:
import CoretoCloudTable from '/snippets/_core-to-cloud-guide-table.md';
-This guide shares some tips, enhancements, technical adjustments, and frequently asked questions that you may encounter during your move.
-
-## dbt Cloud CLI and dbt Cloud IDE
-**dbt Cloud IDE and dbt Cloud CLI tips**
-- The dbt Cloud IDE has a simplified Git functionality
- - You can create feature branches from the branch configured in the development environment.
- - The dbt Cloud IDE makes it possible view saved but not-committed code changes directly in the IDE
- - You can [format or lint](https://docs.getdbt.com/docs/cloud/dbt-cloud-ide/lint-format) your code with `sqlfluff` or `sqlfmt`. This includes support for adding your custom linting rules.
- - Advanced users who prefer to have more control over their git commands can use the [dbt Cloud CLI](https://docs.getdbt.com/docs/cloud/cloud-cli-installation), allowing users to run dbt commands against their dbt Cloud development environment from their local command line with minimal configuration.
-- Whichever they pick, your users will be getting a powerful development experience backed by dbt Cloud. This means the dbt Cloud CLI and the dbt Cloud IDE enable users to natively [defer to production](https://docs.getdbt.com/docs/cloud/about-cloud-develop-defer#defer-in-dbt-cloud-cli) metadata directly in their development workflows. This reduces the number of objects in - The dbt Cloud CLI and dbt Cloud IDE are designed to support [safe parallel execution](https://docs.getdbt.com/reference/dbt-commands#parallel-execution) of dbt commands, leveraging dbt Cloud's infrastructure and its comprehensive [features](https://docs.getdbt.com/docs/cloud/about-cloud/dbt-cloud-features). In contrast, `dbt-core` *doesn't support* safe parallel execution for multiple invocations in the same process.
-- **Caveats**
+## What you'll learn
+If you're reading this guide, you may have already started your move to dbt Cloud and are looking for tips to help you optimize your dbt Cloud experience, including tips and caveats for the following areas:
+
+- [Adapters and connections](https://docs.getdbt.com/guides/core-to-cloud-3?step=3)
+- [Development tools](https://docs.getdbt.com)/guides/core-to-cloud-3?step=4)
+- [Orchestration](https://docs.getdbt.com/guides/core-to-cloud-3?step=5)
+- [dbt Mesh](https://docs.getdbt.com/guides/core-to-cloud-3?step=6)
+- [dbt Semantic Layer](https://docs.getdbt.com/guides/core-to-cloud-3?step=7)
+- [dbt Explorer](https://docs.getdbt.com/guides/core-to-cloud-3?step=8)
+- [What's next?](https://docs.getdbt.com/guides/core-to-cloud-3?step=9)
+
+## Adapters and connections
+
+In dbt Cloud, you can natively connect to your data platform and test its [connection](/docs/connect-adapters) with a click of a button. This is especially useful for users who are new to dbt Cloud or are looking to streamline their connection setup. Here are some tips and caveats to consider:
+
+### Tips
+- Manage [dbt versions](/docs/dbt-versions/upgrade-dbt-version-in-cloud) and ensure team synchronization with dbt Cloud's one-click feature, eliminating the hassle of manual updates and version discrepancies.
+- dbt Cloud supports a whole host of [cloud providers](/docs/cloud/connect-data-platform/about-connections), including Snowflake, Databricks, BigQuery, Fabric, and Redshift (to name a few).
+- Use [Extended Attributes](/docs/deploy/deploy-environments#extended-attributes) to set a flexible [profiles.yml](/docs/core/connect-data-platform/profiles.yml) snippet in your dbt Cloud Environment settings. It gives you more control over environments (both deployment and development) and extends how dbt Cloud connects to the data platform within a given environment.
+ - For example, if you have a field in your `profiles.yml` that you’d like to add to the dbt Cloud adapter user interface, you can use Extended Attributes to set it.
+
+### Caveats
+- Not all parameters are available for adapters.
+- One project can only use one warehouse type.
+
+## Development tools
+
+dbt Cloud empowers data practitioners to develop in the tool of their choice. It ships with a [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) (local) or [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) (browser-based) to build, test, run, and version control your dbt projects.
+
+Here are some tips and caveats to consider when using dbt Cloud's development tools:
+
+### Tips
+
+- The dbt Cloud IDE has a simplified Git functionality:
+ - Create feature branches from the branch configured in the development environment.
+ - View saved but not-committed code changes directly in the IDE.
+ - [Format or lint](/docs/cloud/dbt-cloud-ide/lint-format) your code with `sqlfluff` or `sqlfmt`. This includes support for adding your custom linting rules.
+- Advanced users who prefer to have more control over their Git commands can use the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation), allowing users to run dbt commands against their dbt Cloud development environment from their local command line with minimal configuration.
+- Allows users to natively [defer to production](/docs/cloud/about-cloud-develop-defer#defer-in-dbt-cloud-cli) metadata directly in their development workflows, reducing the number of objects.
+- Support running multiple dbt commands at the same time through [safe parallel execution](/reference/dbt-commands#parallel-execution), a [feature](/docs/cloud/about-cloud/dbt-cloud-features) available in dbt Cloud's infrastructure. In contrast, `dbt-core` *doesn't support* safe parallel execution for multiple invocations in the same process.
+
+### Caveats
- Pre-commit for automated checks before *committing* code is not available (yet).
-- Mass-generating files / interacting with the file system are not available
-- Combining/piping commands, such as `dbt run -s (bash command)` is not available
+- Mass-generating files / interacting with the file system are not available.
+- Combining/piping commands, such as `dbt run -s (bash command)` is not available.
## Orchestration
-**dbt Cloud job scheduler tips**
-- Enable [partial parsing](https://docs.getdbt.com/docs/deploy/deploy-environments#partial-parsing) between jobs in dbt Cloud to significantly speed up project parsing by only processing changed files, optimizing performance for large projects.
-- The dbt Cloud job scheduler can [run multiple CI/CD](https://docs.getdbt.com/docs/deploy/continuous-integration) jobs at the same time, will not block production runs, and stale runs are automatically canceled when a newer commit is pushed. This is because each PR will run in its own schema.
+dbt Cloud provides a robust orchestration that enables users to schedule, run, and monitor dbt jobs with ease. Here are some tips and caveats to consider when using dbt Cloud's orchestration features:
+
+### Tips
+
+- Enable [partial parsing](/docs/deploy/deploy-environments#partial-parsing) between jobs in dbt Cloud to significantly speed up project parsing by only processing changed files, optimizing performance for large projects.
+- [Run multiple CI/CD](/docs/deploy/continuous-integration) jobs at the same time which will not block production runs. The Job scheduler automatically cancels stale runs when a newer commit is pushed. This is because each PR will run in its own schema.
- dbt Cloud automatically cancels a scheduled run if the existing run is still executing. This prevents unnecessary, duplicative executions.
-- Protect you and your data freshness from third-party outages by enabling dbt Cloud’s [Git repository caching](https://docs.getdbt.com/docs/deploy/deploy-environments#git-repository-caching), which keeps a cache of the project's Git repository. (Enterprise only)
-- You can *[chain* deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs#trigger-on-job-completion--) across dbt Cloud projects by configuring your job or using the [Create Job API](https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Create%20Job) to do this. (Team or Enterprise only)
-- [Rerun your jobs](https://docs.getdbt.com/docs/deploy/retry-jobs) from the start or the point of failure if your dbt job run completed with a status of **`Error.`**
-- **Caveats**
+- Protect you and your data freshness from third-party outages by enabling dbt Cloud’s [Git repository caching](/docs/deploy/deploy-environments#git-repository-caching), which keeps a cache of the project's Git repository.
+- [Link deploy jobs](/docs/deploy/deploy-jobs#trigger-on-job-completion--) across dbt Cloud projects by configuring your job or using the [Create Job API](/dbt-cloud/api-v2#/operations/Create%20Job) to do this.
+- [Rerun your jobs](/docs/deploy/retry-jobs) from the start or the point of failure if your dbt job run completed with a status of **`Error.`**
+
+### Caveats
- To store your job configurations as code within a repository, you can:
- Check out our [Terraform provider.](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest/docs/resources/job)
- Alternatively, check out our [jobs-as-code](https://github.com/dbt-labs/dbt-jobs-as-code) repository, which is a tool built to handle dbt Cloud jobs as a well-defined YAML file.
-- dbt Cloud users and external emails can receive notifications if a job fails, succeeds, or is cancelled. To get notifications for warnings, you can create a [webhook subscription](https://docs.getdbt.com/guides/zapier-slack) and post to Slack.
-
-## Adapters and connections
-
-**Adapters tips**
-- In dbt Cloud, you can natively connect to your data platform and test its [connection](https://docs.getdbt.com/docs/connect-adapters) with a click of a button!
-- Manage [dbt versions](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-version-in-cloud) and ensure team synchronization with dbt Cloud's one-click feature, eliminating the hassle of manual updates and version discrepancies.
-- dbt Cloud supports a whole host of cloud providers, including Snowflake, Databricks, BigQuery, Fabric, and Redshift (to name a few). Check out the full list of supported adapters [here](https://docs.getdbt.com/docs/cloud/connect-data-platform/about-connections).
-- Use [Extended Attributes](https://docs.getdbt.com/docs/deploy/deploy-environments#extended-attributes) to set a flexible [profiles.yml](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) snippet in your dbt Cloud Environment settings. It gives you more control over environments (both deployment and development) and extends how dbt Cloud connects to the data platform within a given environment.
- - For example, if you have a field in your `profiles.yml` that you’d like to add to the dbt Cloud adapter user interface, you can use Extended Attributes to set it.
-**Caveats**
-- Not all parameters are available for adapters.
-- One project can only use one warehouse type.
+- dbt Cloud users and external emails can receive notifications if a job fails, succeeds, or is cancelled. To get notifications for warnings, you can create a [webhook subscription](/guides/zapier-slack) and post to Slack.
## dbt Mesh
-To use [cross-project references](https://docs.getdbt.com/docs/collaborate/govern/project-dependencies#how-to-write-cross-project-ref), all developers need to develop with dbt Cloud (either with the dbt Cloud CLI or dbt Cloud IDE). Cross-project references are not supported in dbt Core.
+Use [dbt Mesh](/best-practices/how-we-mesh/mesh-1-intro) to seamlessly integrate and navigate between different projects and models with [cross-project dependencies](/docs/collaborate/govern/project-dependencies#how-to-write-cross-project-ref), enhancing collaboration and data governance. Here are some tips and caveats to consider when using dbt Mesh:
-**dbt Mesh tips**
-- Use [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro) to seamlessly integrate and navigate between different projects and models with [cross-project dependencies](https://docs.getdbt.com/docs/collaborate/govern/project-dependencies#how-to-write-cross-project-ref), enhancing collaboration and data governance.
+### Tips
+- To use [cross-project references](/docs/collaborate/govern/project-dependencies#how-to-write-cross-project-ref), all developers need to develop with dbt Cloud (either with the dbt Cloud CLI or dbt Cloud IDE). Cross-project references are not supported in dbt Core.
- Link models across projects for a modular and scalable approach for your project and teams.
- Manage access to your dbt models both within and across projects using:
- - **[Groups](https://docs.getdbt.com/docs/collaborate/govern/model-access#groups)** - Organize nodes in your dbt DAG that share a logical connection and assign an owner to the entire group.
- - **[Access](https://docs.getdbt.com/docs/collaborate/govern/model-access#access-modifiers)** - Control who can reference models.
- - **[Model Versions](https://docs.getdbt.com/docs/collaborate/govern/model-versions)** - Enable adoption and deprecation of models as they evolve.
- - **[Model Contracts](https://docs.getdbt.com/docs/collaborate/govern/model-contracts)** -Set clear expectations on the shape of the data to ensure data changes upstream of dbt or within a project's logic don't break downstream consumers' data products.
- - Use [dbt-meshify](https://github.com/dbt-labs/dbt-meshify) if you’d separate your mono-repo in order to move to dbt Cloud.
-**Caveat**
+ - **[Groups](/docs/collaborate/govern/model-access#groups)** - Organize nodes in your dbt DAG that share a logical connection and assign an owner to the entire group.
+ - **[Access](/docs/collaborate/govern/model-access#access-modifiers)** - Control who can reference models.
+ - **[Model Versions](/docs/collaborate/govern/model-versions)** - Enable adoption and deprecation of models as they evolve.
+ - **[Model Contracts](/docs/collaborate/govern/model-contracts)** -Set clear expectations on the shape of the data to ensure data changes upstream of dbt or within a project's logic don't break downstream consumers' data products.
+- Use [dbt-meshify](https://github.com/dbt-labs/dbt-meshify) to separate your mono-repo and move to dbt Cloud.
+
+### Caveat
- Currently, dbt Mesh does not support circular project dependencies.
-## dbt Semantic Layer
-
-**dbt Semantic Layer tips**
-- Leverage the [dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-sl), powered by MetricFlow, to create a unified view of your business metrics, ensuring consistency across all analytics tools.
-- Define semantic models and metrics once with MetricFlow, and reuse them across various analytics platforms, reducing redundancy and errors.
-- Use the dbt Semantic Layer APIs to query metrics in downstream tools for consistent, reliable data metrics.
-- Connect to several data applications, from business intelligence tools to notebooks, spreadsheets, data catalogs, and more, to query your metrics. Available integrations include Tableau, Google Sheets, Hex, and more!
-- Use exports to materialize commonly used queries directly within your data platform, on a schedule.
-**Caveats**
+ ## dbt Semantic Layer
+
+Leverage the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl), powered by MetricFlow, to create a unified view of your business metrics, ensuring consistency across all analytics tools. Here are some tips and caveats to consider when using dbt Semantic Layer:
+
+### Tips
+- Define semantic models and metrics once in dbt Cloud with the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) (powered by MetricFlow). Reuse them across various analytics platforms, reducing redundancy and errors.
+- Use the [dbt Semantic Layer APIs](/docs/dbt-cloud-apis/sl-api-overview) to query metrics in downstream tools for consistent, reliable data metrics.
+- Connect to several data applications, from business intelligence tools to notebooks, spreadsheets, data catalogs, and more, to query your metrics. [Available integrations](/docs/use-dbt-semantic-layer/avail-sl-integrations) include Tableau, Google Sheets, Hex, and more.
+- Use [exports](/docs/use-dbt-semantic-layer/exports) to write commonly used queries directly within your data platform, on a schedule.
+
+### Caveats
- dbt Semantic Layer currently supports the Deployment environment for querying. Development querying experience coming soon.
-- You can run queries/semantic layer commands in the dbt Cloud CLI, however running queries/semantic layer commands in the dbt Cloud IDE isn’t supported *yet.*
- - Note that SSH tunneling for [Postgres and Redshift](https://docs.getdbt.com/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb) connections, [PrivateLink](https://docs.getdbt.com/docs/cloud/secure/about-privatelink), and [Single sign-on (SSO)](https://docs.getdbt.com/docs/cloud/manage-access/sso-overview) isn’t supported in the dbt Semantic Layer *yet*.
+- Run queries/semantic layer commands in the dbt Cloud CLI, however running queries/semantic layer commands in the dbt Cloud IDE isn’t supported *yet.*
+- dbt Semantic Layer doesn't yet support SSH tunneling for [Postgres or Redshift](/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb) connections. It also doesn't support using [Single sign-on (SSO)](/docs/cloud/manage-access/sso-overview) for Semantic Layer [production credentials](/docs/dbt-cloud-apis/service-tokens#permissions-for-service-account-tokens), however, SSO is supported for development user accounts.
## dbt Explorer
-**dbt Explorer tips**
-- [dbt Explorer](https://docs.getdbt.com/docs/collaborate/explore-projects) enhances your ability to discover and understand your data models through rich metadata and lineage visualization.
+[dbt Explorer](/docs/collaborate/explore-projects) enhances your ability to discover and understand your data models through rich metadata and lineage visualization. Here are some tips and caveats to consider when using dbt Explorer:
+
+### Tips
- Improve documentation accessibility, making it easier for teams to find and understand the data assets available to them.
- Use the search and filter capabilities in dbt Explorer to quickly locate models, sources, and tests, streamlining your workflow.
-- View all the different projects and public models in the account, where the public models are defined, and how they are used to gain a better understanding of your cross-project resources.
-- Access column-level lineage (CLL) for the resources in your dbt project. (Enterprise plan only).
+- View all the [different projects](/docs/collaborate/explore-multiple-projects) and public models in the account, where the public models are defined, and how they are used to gain a better understanding of your cross-project resources.
+- Use the [Lenses](/docs/collaborate/explore-projects#lenses) feature, which are like a map layers for your DAG, available from your project's lineage graph. Lenses help you further understand your project’s contextual metadata at scale, especially to distinguish a particular model or a subset of models.
+- Access column-level lineage (CLL) for the resources in your dbt project.
-**Caveats**
+### Caveats
- There has been at least one successful job run in the production deployment environment.
- Familiarize yourself with dbt Explorer’s features to fully leverage its capabilities to avoid missed opportunities for efficiency gains.
## What's next?
-Congratulations on making it through the Move from dbt Core to dbt Cloud guide 🎉! We hope you’re equipped with useful insights and tips to help you with your move. Something to note is that moving from dbt Core to dbt Cloud isn’t just about evolving your data projects — it's about unlocking new levels of collaboration, governance, efficiency, and innovation within your team.
+
+Congratulations on making it through the guide 🎉!
+
+We hope you’re equipped with useful insights and tips to help you with your move. Something to note is that moving from dbt Core to dbt Cloud isn’t just about evolving your data projects — it's about exploring new levels of collaboration, governance, efficiency, and innovation within your team.
For next steps, you can continue exploring our 3-part-guide series on moving from dbt Core to dbt Cloud:
@@ -115,20 +147,20 @@ For next steps, you can continue exploring our 3-part-guide series on moving fro
-## Support
+### Resources
If you need any additional help or have some questions, you can reach out to us in the following ways:
-- Our [Support team](https://docs.getdbt.com/docs/dbt-support)
-- The dbt Slack community. We have specific slack channels dedicated to different product surface areas.
+- [dbt Cloud learn courses](https://courses.getdbt.com/collections) for on-demand video learning.
+- Our [Support team](https://docs.getdbt.com/docs/dbt-support) is always available to help you troubleshoot your dbt Cloud issues.
+- Join the [dbt Community](https://community.getdbt.com/) to connect with other dbt users, ask questions, and share best practices.
- Subscribe to the [dbt Cloud RSS alerts](https://status.getdbt.com/)
-- Your Account representative
+- Enterprise accounts have an account management team available to help troubleshoot solutions and account management assistance. [Book a demo](https://www.getdbt.com/contact) to learn more.
....we’re happy to help!
-## Resources
-
For tailored assistance, you can use the following resources:
- Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights
- Work with the [dbt Labs’ Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and move.
+
From 337b146a439c27ad105d68df9289070f600313e8 Mon Sep 17 00:00:00 2001
From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com>
Date: Fri, 5 Apr 2024 12:00:27 +0100
Subject: [PATCH 029/131] Update core-to-cloud-1.md
---
website/docs/guides/core-to-cloud-1.md | 17 +++++++----------
1 file changed, 7 insertions(+), 10 deletions(-)
diff --git a/website/docs/guides/core-to-cloud-1.md b/website/docs/guides/core-to-cloud-1.md
index 45057ef5937..63185bdca7d 100644
--- a/website/docs/guides/core-to-cloud-1.md
+++ b/website/docs/guides/core-to-cloud-1.md
@@ -36,6 +36,12 @@ Learn more about [dbt Cloud features](/docs/cloud/about-cloud/dbt-cloud-feature
dbt Core is an open-source tool that enables data teams to define and execute data transformations in a cloud data warehouse following analytics engineering best practices. While this can work well for ‘single players’ and small technical teams, all development happens on a command-line interface, and production deployments must be self-hosted and maintained. This requires significant, costly work that adds up over time to maintain and scale.
+## Prerequisites
+
+- You have an existing dbt Core project connected to a Git repository and data platform supported in [dbt Cloud](/docs/cloud/connect-data-platform/about-connections).
+- A supported version of dbt or select [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) of dbt.
+- You have a dbt Cloud account. **[Don't have one? Start your free trial today](https://www.getdbt.com/signup)**!
+
### What you'll learn
This guide outlines the steps you need to take to move from dbt Core to dbt Cloud and highlights the necessary technical changes:
@@ -49,18 +55,9 @@ This guide outlines the steps you need to take to move from dbt Core to dbt Clou
- [Models configuration](https://docs.getdbt.com/guides/core-to-cloud-1?step=9#models-configuration): Get insights on validating and running your models in dbt Cloud, using either the dbt Cloud IDE or dbt Cloud CLI.
- [What's next?](https://docs.getdbt.com/guides/core-to-cloud-1?step=10#whats-next): Summarizes key takeaways and introduces what to expect in the following guides.
-If you are hosting dbt Core yourself, you need to answer these questions too. dbt Cloud provides a toolbox of options with sensible out-of-the-box defaults so you can focus on your data and analytics engineering workflows.
-
-## Prerequisites
-
-- You have an existing dbt Core project connected to a Git repository and data platform supported in [dbt Cloud](/docs/cloud/connect-data-platform/about-connections).
-- A supported version of dbt or select [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) of dbt.
-- You have a dbt Cloud account. **[Don't have one? Start your free trial today](https://www.getdbt.com/signup)**!
### Related docs
-- [Learn dbt Cloud](https://courses.getdbt.com/collections)
-- [Develop with dbt Cloud](/docs/cloud/about-develop-dbt)
-- [Deploy jobs](/docs/deploy/deployments)
+- [Learn dbt Cloud](https://courses.getdbt.com/collections) on-demand video learning.
- Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights
- Work with the [dbt Labs’ Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and migration.
From b27712a3b173acf074aa8dabe9b0d7f988b974c3 Mon Sep 17 00:00:00 2001
From: mirnawong1
Date: Fri, 5 Apr 2024 12:00:39 +0100
Subject: [PATCH 030/131] add links
---
website/docs/docs/core/about-core-setup.md | 5 ++++-
website/docs/docs/core/installation-overview.md | 5 ++++-
2 files changed, 8 insertions(+), 2 deletions(-)
diff --git a/website/docs/docs/core/about-core-setup.md b/website/docs/docs/core/about-core-setup.md
index 16bfe18fc37..9a2a5e69be5 100644
--- a/website/docs/docs/core/about-core-setup.md
+++ b/website/docs/docs/core/about-core-setup.md
@@ -19,4 +19,7 @@ If you're interested in using a command line interface to [develop dbt projects
If you need a more detailed first-time setup guide for specific data platforms, read our [quickstart guides](https://docs.getdbt.com/guides).
## Related docs
-- [Move from dbt Core to dbt Cloud](/guides/core-to-cloud-1?step=1)
+- Refer to the 3-part guides series for details, considerations, and tips on how to move to dbt Cloud:
+ - [Move from dbt Core to dbt Cloud: Get started](/guides/core-to-cloud-1?step=1)
+ - [Move from dbt Core to dbt Cloud: What you need to know](/guides/core-to-cloud-2?step=1)
+ - [Move from dbt Core to dbt Cloud: Optimization tips](/guides/core-to-cloud-3?step=1)
diff --git a/website/docs/docs/core/installation-overview.md b/website/docs/docs/core/installation-overview.md
index 2e112066048..789ca1da4ff 100644
--- a/website/docs/docs/core/installation-overview.md
+++ b/website/docs/docs/core/installation-overview.md
@@ -52,4 +52,7 @@ Most command-line tools, including dbt, have a `--help` flag that you can use to
:::
## Related docs
-- [Move from dbt Core to dbt Cloud](/guides/core-to-cloud-1?step=1)
+- Refer to the 3-part guides series for details, considerations, and tips on how to move to dbt Cloud:
+ - [Move from dbt Core to dbt Cloud: Get started](/guides/core-to-cloud-1?step=1)
+ - [Move from dbt Core to dbt Cloud: What you need to know](/guides/core-to-cloud-2?step=1)
+ - [Move from dbt Core to dbt Cloud: Optimization tips](/guides/core-to-cloud-3?step=1)
From 542bd1d205e527d068e7e8c88973ea91769c5737 Mon Sep 17 00:00:00 2001
From: mirnawong1
Date: Fri, 5 Apr 2024 12:01:58 +0100
Subject: [PATCH 031/131] add link
---
website/docs/guides/core-to-cloud-2.md | 1 +
website/docs/guides/core-to-cloud-3.md | 4 +---
2 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/website/docs/guides/core-to-cloud-2.md b/website/docs/guides/core-to-cloud-2.md
index 214352410bc..a2cca26621f 100644
--- a/website/docs/guides/core-to-cloud-2.md
+++ b/website/docs/guides/core-to-cloud-2.md
@@ -191,5 +191,6 @@ For next steps, you can continue exploring our 3-part-guide series on moving fro
- [Learn dbt Cloud](https://courses.getdbt.com/collections) courses
- Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights
- Work with the [dbt Labs’ Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and migration.
+- [How dbt CLou compares with dbt Core](https://www.getdbt.com/product/dbt-core-vs-dbt-cloud) for a detailed comparison of dbt Core and dbt Cloud.
diff --git a/website/docs/guides/core-to-cloud-3.md b/website/docs/guides/core-to-cloud-3.md
index 084a66a3708..29a0c5c9cea 100644
--- a/website/docs/guides/core-to-cloud-3.md
+++ b/website/docs/guides/core-to-cloud-3.md
@@ -156,11 +156,9 @@ If you need any additional help or have some questions, you can reach out to us
- Join the [dbt Community](https://community.getdbt.com/) to connect with other dbt users, ask questions, and share best practices.
- Subscribe to the [dbt Cloud RSS alerts](https://status.getdbt.com/)
- Enterprise accounts have an account management team available to help troubleshoot solutions and account management assistance. [Book a demo](https://www.getdbt.com/contact) to learn more.
-
-....we’re happy to help!
+- [How dbt CLou compares with dbt Core](https://www.getdbt.com/product/dbt-core-vs-dbt-cloud) for a detailed comparison of dbt Core and dbt Cloud.
For tailored assistance, you can use the following resources:
- Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights
- Work with the [dbt Labs’ Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and move.
-
From f966758e906eba7d23c90df3fea89a98e5d2abc9 Mon Sep 17 00:00:00 2001
From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com>
Date: Fri, 5 Apr 2024 12:02:23 +0100
Subject: [PATCH 032/131] Update core-to-cloud-1.md
---
website/docs/guides/core-to-cloud-1.md | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/website/docs/guides/core-to-cloud-1.md b/website/docs/guides/core-to-cloud-1.md
index 63185bdca7d..924f7ad6939 100644
--- a/website/docs/guides/core-to-cloud-1.md
+++ b/website/docs/guides/core-to-cloud-1.md
@@ -258,7 +258,7 @@ You have learned:
- How to configure your development, orchestration, and CI/CD environments
- How to set up environment variables and validate your models
-For next steps, you can continue exploring our 3-part-guide series on moving from dbt Core to dbt Cloud:
+For the next steps, you can continue exploring our 3-part-guide series on moving from dbt Core to dbt Cloud:
@@ -266,7 +266,8 @@ For next steps, you can continue exploring our 3-part-guide series on moving fro
- [Learn dbt Cloud](https://courses.getdbt.com/collections) video courses for on-demand learning.
- Book [expert-led demos](https://www.getdbt.com/resources/dbt-cloud-demos-with-experts) and insights.
- Work with the [dbt Labs’ Professional Services](https://www.getdbt.com/dbt-labs/services) team to support your data organization and migration.
-
+- [How dbt CLou compares with dbt Core](https://www.getdbt.com/product/dbt-core-vs-dbt-cloud) for a detailed comparison of dbt Core and dbt Cloud.
+-
From d5c26872a4ade8ec54d4d5fbc20d081c5fd9f73a Mon Sep 17 00:00:00 2001
From: mirnawong1
Date: Fri, 5 Apr 2024 12:02:49 +0100
Subject: [PATCH 033/131] add next
---
website/docs/guides/core-to-cloud-2.md | 4 ++--
website/docs/guides/core-to-cloud-3.md | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/website/docs/guides/core-to-cloud-2.md b/website/docs/guides/core-to-cloud-2.md
index a2cca26621f..3a85f3d00d1 100644
--- a/website/docs/guides/core-to-cloud-2.md
+++ b/website/docs/guides/core-to-cloud-2.md
@@ -181,9 +181,9 @@ This guide should now have given you some insight and equipped you with a framew
-Congratulations on finishing this guide, we hope it's given you insight into the considerations you need to take to best plan your move to dbt Cloud.
+Congratulations on finishing this guide, we hope it's given you insight into the considerations you need to take to best plan your move to dbt Cloud.
-For next steps, you can continue exploring our 3-part-guide series on moving from dbt Core to dbt Cloud:
+For the next steps, you can continue exploring our 3-part-guide series on moving from dbt Core to dbt Cloud:
diff --git a/website/docs/guides/core-to-cloud-3.md b/website/docs/guides/core-to-cloud-3.md
index 29a0c5c9cea..73d4f7b9b4a 100644
--- a/website/docs/guides/core-to-cloud-3.md
+++ b/website/docs/guides/core-to-cloud-3.md
@@ -141,7 +141,7 @@ Congratulations on making it through the guide 🎉!
We hope you’re equipped with useful insights and tips to help you with your move. Something to note is that moving from dbt Core to dbt Cloud isn’t just about evolving your data projects — it's about exploring new levels of collaboration, governance, efficiency, and innovation within your team.
-For next steps, you can continue exploring our 3-part-guide series on moving from dbt Core to dbt Cloud:
+For the next steps, you can continue exploring our 3-part-guide series on moving from dbt Core to dbt Cloud:
From 1a92c34122a191950a9a60cb2948b2f56c66321a Mon Sep 17 00:00:00 2001
From: mirnawong1
Date: Fri, 5 Apr 2024 12:27:55 +0100
Subject: [PATCH 034/131] jerco's feedback
---
website/docs/guides/core-to-cloud-3.md | 38 +++++++++++++++++++++-----
1 file changed, 31 insertions(+), 7 deletions(-)
diff --git a/website/docs/guides/core-to-cloud-3.md b/website/docs/guides/core-to-cloud-3.md
index 73d4f7b9b4a..4cec256445c 100644
--- a/website/docs/guides/core-to-cloud-3.md
+++ b/website/docs/guides/core-to-cloud-3.md
@@ -50,23 +50,47 @@ In dbt Cloud, you can natively connect to your data platform and test its [conn
dbt Cloud empowers data practitioners to develop in the tool of their choice. It ships with a [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) (local) or [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) (browser-based) to build, test, run, and version control your dbt projects.
-Here are some tips and caveats to consider when using dbt Cloud's development tools:
+Both development tools are tailored to suit different roles and preferences within your team. To streamline your team’s workflow, it's important to know who is likely to prefer the dbt Cloud IDE and who might lean towards the dbt Cloud CLI. This section is a little different to the others and aims to clarify these preferences.
-### Tips
+### dbt Cloud IDE
+A single web-based interface for building, testing, running, and version-controlling dbt projects. It compiles dbt code into SQL and executes it directly on your database.
+
+**Who might prefer the dbt Cloud IDE?**
+
+- New dbt users or those transitioning from other tools who appreciate a more guided experience through a browser-based interface.
+- Team members focused on development speed and convenience, including analysts and less technical stakeholders.
+- Individuals who prioritize direct feedback from the IDE, such as seeing unsaved changes.
+
+**Key features**
- The dbt Cloud IDE has a simplified Git functionality:
- Create feature branches from the branch configured in the development environment.
- View saved but not-committed code changes directly in the IDE.
- - [Format or lint](/docs/cloud/dbt-cloud-ide/lint-format) your code with `sqlfluff` or `sqlfmt`. This includes support for adding your custom linting rules.
-- Advanced users who prefer to have more control over their Git commands can use the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation), allowing users to run dbt commands against their dbt Cloud development environment from their local command line with minimal configuration.
+- [Format or lint](/docs/cloud/dbt-cloud-ide/lint-format) your code with `sqlfluff` or `sqlfmt`. This includes support for adding your custom linting rules.
- Allows users to natively [defer to production](/docs/cloud/about-cloud-develop-defer#defer-in-dbt-cloud-cli) metadata directly in their development workflows, reducing the number of objects.
- Support running multiple dbt commands at the same time through [safe parallel execution](/reference/dbt-commands#parallel-execution), a [feature](/docs/cloud/about-cloud/dbt-cloud-features) available in dbt Cloud's infrastructure. In contrast, `dbt-core` *doesn't support* safe parallel execution for multiple invocations in the same process.
-### Caveats
+The dbt Cloud IDE provides a simplified interface that's accessible to all users, regardless of their technical background. However, there are some capabilities that are intentionally not available in the dbt Cloud IDE due to its focus on simplicity and audience:
+
- Pre-commit for automated checks before *committing* code is not available (yet).
- Mass-generating files / interacting with the file system are not available.
- Combining/piping commands, such as `dbt run -s (bash command)` is not available.
-
+
+### dbt Cloud CLI
+The dbt Cloud CLI allows you to run dbt commands against your dbt Cloud development environment from your local command line. For users who seek full control over their development environment and ideal for those comfortable with the command line.
+
+**Who might prefer the dbt Cloud CLI?**
+
+- Experienced dbt users and developers looking for granular control over their Git workflows.
+- Data engineers and practitioners who need to perform complex operations, like mass file generation or specific command combinations.
+
+**Key features**
+
+- Allows users to run dbt commands against their dbt Cloud development environment from their local command line with minimal configuration.
+- Allows users to natively [defer to production](/docs/cloud/about-cloud-develop-defer#defer-in-dbt-cloud-cli) metadata directly in their development workflows, reducing the number of objects.
+- Support running multiple dbt commands at the same time through [safe parallel execution](/reference/dbt-commands#parallel-execution), a [feature](/docs/cloud/about-cloud/dbt-cloud-features) available in dbt Cloud's infrastructure. In contrast, `dbt-core` *doesn't support* safe parallel execution for multiple invocations in the same process.
+- Able to use Visual Studio (VS) Code extensions (such as [dbt-power-user](https://marketplace.visualstudio.com/items?itemName=innoverio.vscode-dbt-power-user) to enhance the development experience by adding extra functionalities.
+
## Orchestration
dbt Cloud provides a robust orchestration that enables users to schedule, run, and monitor dbt jobs with ease. Here are some tips and caveats to consider when using dbt Cloud's orchestration features:
@@ -81,7 +105,7 @@ dbt Cloud provides a robust orchestration that enables users to schedule, run, a
- [Rerun your jobs](/docs/deploy/retry-jobs) from the start or the point of failure if your dbt job run completed with a status of **`Error.`**
### Caveats
-- To store your job configurations as code within a repository, you can:
+- To automate the setup and configuration of your dbt Cloud platform, you can store your job configurations as code within a repository:
- Check out our [Terraform provider.](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest/docs/resources/job)
- Alternatively, check out our [jobs-as-code](https://github.com/dbt-labs/dbt-jobs-as-code) repository, which is a tool built to handle dbt Cloud jobs as a well-defined YAML file.
- dbt Cloud users and external emails can receive notifications if a job fails, succeeds, or is cancelled. To get notifications for warnings, you can create a [webhook subscription](/guides/zapier-slack) and post to Slack.
From 2dd8f68427731a7a8b2a16603cd6fbdd6328466e Mon Sep 17 00:00:00 2001
From: mirnawong1
Date: Fri, 5 Apr 2024 12:29:15 +0100
Subject: [PATCH 035/131] update
---
website/docs/guides/core-to-cloud-2.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/guides/core-to-cloud-2.md b/website/docs/guides/core-to-cloud-2.md
index 3a85f3d00d1..39dabd51214 100644
--- a/website/docs/guides/core-to-cloud-2.md
+++ b/website/docs/guides/core-to-cloud-2.md
@@ -1,5 +1,5 @@
---
-title: 'Move from dbt Core to dbt Cloud: Everything you need to know'
+title: 'Move from dbt Core to dbt Cloud: What you need to know'
id: core-to-cloud-2
description: "Use this guide to understand the considerations and methods you need to move from dbt Core to dbt Cloud."
hoverSnippet: "Use this guide to understand the considerations and methods you need to move from dbt Core to dbt Cloud."
From 07985700150eaa92e5c7e27725569b37f2f47b70 Mon Sep 17 00:00:00 2001
From: mirnawong1
Date: Fri, 5 Apr 2024 12:31:38 +0100
Subject: [PATCH 036/131] fix link
---
website/docs/guides/core-to-cloud-1.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/docs/guides/core-to-cloud-1.md b/website/docs/guides/core-to-cloud-1.md
index 0028d4c61fb..ad82eb12068 100644
--- a/website/docs/guides/core-to-cloud-1.md
+++ b/website/docs/guides/core-to-cloud-1.md
@@ -241,7 +241,7 @@ You have learned:
- How to configure your development, orchestration, and CI/CD environments
- How to set up environment variables and validate your models.
-Review the next guide, [Move from dbt Core to dbt Cloud: Everything you need to know](/docs/guides/core-to-cloud-2), in this series to learn about the strategic considerations you need to take to move from dbt Core to dbt Cloud.
+Review the next guide, [Move from dbt Core to dbt Cloud: Everything you need to know](/guides/core-to-cloud-2), in this series to learn about the strategic considerations you need to take to move from dbt Core to dbt Cloud.