From 4273d57b9db228b73e47870e74e02af7944048ff Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Tue, 6 Aug 2024 10:27:22 +0100 Subject: [PATCH] add as snippet --- .../how-we-mesh/mesh-2-who-is-dbt-mesh-for.md | 8 ++++++-- .../how-we-mesh/mesh-3-structures.md | 16 +++------------- .../best-practices/how-we-mesh/mesh-5-faqs.md | 12 ++---------- .../collaborate/govern/project-dependencies.md | 15 +++------------ website/snippets/_mesh-cycle-detection.md | 14 ++++++++++++++ 5 files changed, 28 insertions(+), 37 deletions(-) create mode 100644 website/snippets/_mesh-cycle-detection.md diff --git a/website/docs/best-practices/how-we-mesh/mesh-2-who-is-dbt-mesh-for.md b/website/docs/best-practices/how-we-mesh/mesh-2-who-is-dbt-mesh-for.md index e57ed5c966f..b6fadc2d7a6 100644 --- a/website/docs/best-practices/how-we-mesh/mesh-2-who-is-dbt-mesh-for.md +++ b/website/docs/best-practices/how-we-mesh/mesh-2-who-is-dbt-mesh-for.md @@ -20,8 +20,12 @@ Is dbt Mesh a good fit in this scenario? Absolutely! There is no other way to sh ### Adoption challenges -- Onboarding hundreds of people and dozens of projects is full of friction! The challenges of a scaled, global organization are not to be underestimated. To start the migration, prioritize teams that have strong dbt familiarity and fundamentals. dbt Mesh is an advancement of core dbt deployments, so these teams are likely to have a smoother transition. Additionally, prioritize teams that manage strategic data assets that need to be shared widely. This ensures that dbt Mesh will help your teams deliver concrete value quickly. -- Bi-directional project dependencies -- currently, projects in dbt Mesh are treated like dbt resources in that they cannot depend on each other. In reality, domain teams likely need the ability to have “chatty” APIs; otherwise, they need to split projects beyond the 1:1 mapping with team boundaries. While this constraint exists today, we're working to remove this point of friction. More information about this will be provided in the near future! +- Onboarding hundreds of people and dozens of projects is full of friction! The challenges of a scaled, global organization are not to be underestimated. To start the migration, prioritize teams that have strong dbt familiarity and fundamentals. dbt Mesh is an advancement of core dbt deployments, so these teams are likely to have a smoother transition. + + Additionally, prioritize teams that manage strategic data assets that need to be shared widely. This ensures that dbt Mesh will help your teams deliver concrete value quickly. +- Bi-directional project dependencies -- currently, projects in dbt Mesh are treated like dbt resources in that they cannot depend on each other. However, many teams may want to be able to share data assets back and forth between teams. + + We've added support for [enabling bidirectional dependencies](/best-practices/how-we-mesh/mesh-3-structures#cycle-detection) across projects. If this sounds like your organization, dbt Mesh is the architecture you should pursue. ✅ diff --git a/website/docs/best-practices/how-we-mesh/mesh-3-structures.md b/website/docs/best-practices/how-we-mesh/mesh-3-structures.md index e50771762a9..c75c566610b 100644 --- a/website/docs/best-practices/how-we-mesh/mesh-3-structures.md +++ b/website/docs/best-practices/how-we-mesh/mesh-3-structures.md @@ -66,22 +66,12 @@ Since the launch of dbt Mesh, the most common pattern we've seen is one where pr Users may need to contribute models across multiple projects and this is fine. There will be some friction doing this, versus a single repo, but this is _useful_ friction, especially if upstreaming a change from a “spoke” to a “hub.” This should be treated like making an API change, one that the other team will be living with for some time to come. You should be concerned if your teammates find they need to make a coordinated change across multiple projects very frequently (every week), or as a key prerequisite for ~20%+ of their work. -### Cycle detection +### Cycle detection -Currently, the default behavior for "project" dependencies enforces that these relationships only go in one direction, meaning that the `jaffle_finance` project could not add a new model that depends, on any public models produced by the `jaffle_marketing` project. dbt will check for cycles across projects and raise errors if any are detected. +import CycleDetection from '/snippets/_mesh-cycle-detection.md'; -However, many teams may want to be able to share data assets back and forth between teams. _We've added support for enabling bidirectional dependencies across projects, currently in beta_. + -To enable this in your account, set the environment variable `DBT_CLOUD_PROJECT_CYCLES_ALLOWED` to `TRUE` in all your dbt Cloud environments. This allows you to create bidirectional dependencies between projects, so long as the new dependency does not introduce any node-level cycles. - -When setting up projects that depend on each other, it's important to do so in a stepwise fashion. Each project must run and produce public models before the original producer project can take a dependency on the original consumer project. For example, the order of operations would be as follows for a simple two-project setup: - -1. The `project_a` project runs in a deployment environment and produces public models. -2. The `project_b` project adds `project_a` as a dependency. -3. The `project_b` project runs in a deployment environment and produces public models. -4. The `project_a` project adds `project_b` as a dependency. - -If you enable this feature and experience any issues, please reach out to dbt Cloud support. ### Tips and tricks diff --git a/website/docs/best-practices/how-we-mesh/mesh-5-faqs.md b/website/docs/best-practices/how-we-mesh/mesh-5-faqs.md index 2292567ba4c..f52cff78188 100644 --- a/website/docs/best-practices/how-we-mesh/mesh-5-faqs.md +++ b/website/docs/best-practices/how-we-mesh/mesh-5-faqs.md @@ -78,18 +78,10 @@ dbt Mesh also enhances the interoperability and reusability of data across diffe -Currently, the default behavior for "project" dependencies enforces that these relationships only go in one direction, meaning that the `jaffle_finance` project could not add a new model that depends, on any public models produced by the `jaffle_marketing` project. dbt will check for cycles across projects and raise errors if any are detected. +import CycleDetection from '/snippets/_mesh-cycle-detection.md'; -However, many teams may want to be able to share data assets back and forth between teams. **We've added support for enabling bidirectional dependencies across projects, currently in beta**. To enable this in your account, simply set the environment variable `DBT_CLOUD_PROJECT_CYCLES_ALLOWED` to `TRUE` in all yur dbt Cloud environments. This will allow you to create bidirectional dependencies between projects, so long as there are not any node-level cycles introduced by the new dependency. + -When setting up projects that depend on each other, it's important to do so in a stepwise fashion. Each project must run and produce public models before the original producer project can take a dependency on the original consumer project. For example, the order of operations would be as follows for a simple two-project setup: - -1. The `project_a` project runs in a deployment environment and produce public models. -2. The `project_b` project adds `project_a` as a dependency. -3. The `project_b` project runs in a deployment environment and produces public models. -4. The `project_a` project adds `project_b` as a dependency. - -If you enable this feature and experience any issues, please be sure to each out to dbt Cloud support. diff --git a/website/docs/docs/collaborate/govern/project-dependencies.md b/website/docs/docs/collaborate/govern/project-dependencies.md index 72ab0c9b13a..a56646b0d0b 100644 --- a/website/docs/docs/collaborate/govern/project-dependencies.md +++ b/website/docs/docs/collaborate/govern/project-dependencies.md @@ -95,20 +95,11 @@ with monthly_revenue as ( -**Cycle detection:** +#### Cycle detection -Currently, the default behavior for "project" dependencies enforces that these relationships only go in one direction, meaning that the `jaffle_finance` project could not add a new model that depends, on any public models produced by the `jaffle_marketing` project. dbt will check for cycles across projects and raise errors if any are detected. +import CycleDetection from '/snippets/_mesh-cycle-detection.md'; -However, many teams may want to be able to share data assets back and forth between teams. **We've added support for enabling bidirectional dependencies across projects, currently in beta**. To enable this in your account, simply set the environment variable `DBT_CLOUD_PROJECT_CYCLES_ALLOWED` to `TRUE` in all yur dbt Cloud environments. This will allow you to create bidirectional dependencies between projects, so long as there are not any node-level cycles introduced by the new dependency. - -When setting up projects that depend on each other, it's important to do so in a stepwise fashion. Each project must run and produce public models before the original producer project can take a dependency on the original consumer project. For example, to enable this in our above example, the order of operations would be as follows: - -1. The `jaffle_finance` project runs in a deployment environment and produce public models. -2. The `jaffle_marketing` project adds `jaffle_finance` as a dependency. -3. The `jaffle_marketing` project runs in a deployment environment and produces public models. -4. The `jaffle_finance` project adds `jaffle_marketing` as a dependency. - -If you enable this feature and experience any issues, please be sure to each out to dbt Cloud support. + For more guidance on how to use dbt Mesh, refer to the dedicated [dbt Mesh guide](/best-practices/how-we-mesh/mesh-1-intro) and also our freely available [dbt Mesh learning course](https://learn.getdbt.com/courses/dbt-mesh). diff --git a/website/snippets/_mesh-cycle-detection.md b/website/snippets/_mesh-cycle-detection.md new file mode 100644 index 00000000000..2a48d0a15bd --- /dev/null +++ b/website/snippets/_mesh-cycle-detection.md @@ -0,0 +1,14 @@ +Currently, the default behavior for "project" dependencies enforces that these relationships only go in one direction, meaning that the `jaffle_finance` project could not add a new model that depends, on any public models produced by the `jaffle_marketing` project. dbt will check for cycles across projects and raise errors if any are detected. + +However, many teams may want to be able to share data assets back and forth between teams. _We've added support for enabling bidirectional dependencies across projects, currently in beta_. + +To enable this in your account, set the environment variable `DBT_CLOUD_PROJECT_CYCLES_ALLOWED` to `TRUE` in all your dbt Cloud environments. This allows you to create bidirectional dependencies between projects, so long as the new dependency does not introduce any node-level cycles. + +When setting up projects that depend on each other, it's important to do so in a stepwise fashion. Each project must run and produce public models before the original producer project can take a dependency on the original consumer project. For example, the order of operations would be as follows for a simple two-project setup: + +1. The `project_a` project runs in a deployment environment and produces public models. +2. The `project_b` project adds `project_a` as a dependency. +3. The `project_b` project runs in a deployment environment and produces public models. +4. The `project_a` project adds `project_b` as a dependency. + +If you enable this feature and experience any issues, please reach out to [dbt Cloud support](mailto:support@getdbt.com).