Skip to content

Commit

Permalink
Merge branch 'current' into add-explorer-course-link
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Aug 8, 2024
2 parents ef94be9 + 7940fa0 commit c22b03e
Show file tree
Hide file tree
Showing 11 changed files with 60 additions and 12 deletions.
4 changes: 4 additions & 0 deletions website/docs/best-practices/how-we-mesh/mesh-5-faqs.md
Original file line number Diff line number Diff line change
Expand Up @@ -264,12 +264,16 @@ Historical limitations required customers to "duplicate" projects so that one ac

<DetailsToggle alt_header="How does the dbt Semantic Layer relate to and work with dbt Mesh?">

import SLMeshFAQs from '/snippets/_sl-dbt-mesh-faq.md';

The [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) and dbt Mesh are complementary mechanisms enabled by dbt Cloud that work together to enhance the management, usability, and governance of data in large-scale data environments.

The Semantic Layer in dbt Cloud allows teams to centrally define business metrics and dimensions. It ensures consistent and reliable metric definitions across various analytics tools and platforms.

dbt Mesh enables organizations to split their data architecture into multiple domain-specific projects, while retaining the ability to reference “public” models across projects. It is also possible to reference a “public” model from another project for the purpose of defining semantic models and metrics. Your organization can have multiple dbt projects feed into a unified semantic layer, ensuring that metrics and dimensions are consistently defined and understood across these domains.

<SLMeshFAQs/>

</DetailsToggle>

<DetailsToggle alt_header="How does dbt Explorer relate to and work with dbt Mesh?">
Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/build/metricflow-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ You can use the `dbt sl` prefix before the command name to execute them in the d
- [`list dimensions`](#list) &mdash; Lists unique dimensions for metrics.
- [`list dimension-values`](#list-dimension-values) &mdash; List dimensions with metrics.
- [`list entities`](#list-entities) &mdash; Lists all unique entities.
- [`list saved queries`](#list-saved-queries) &mdash; Lists available saved queries. Use the `--show-exports` flag to display each export listed under a saved query.
- [`list saved-queries`](#list-saved-queries) &mdash; Lists available saved queries. Use the `--show-exports` flag to display each export listed under a saved query.
- [`query`](#query) &mdash; Query metrics, saved queries, and dimensions you want to see in the command line interface. Refer to [query examples](#query-examples) to help you get started.
- [`export`](#export) &mdash; Runs exports for a singular saved query for testing and generating exports in your development environment. You can also use the `--select` flag to specify particular exports from a saved query.
- [`export-all`](#export-all) &mdash; Runs exports for multiple saved queries at once, saving time and effort.
Expand Down
19 changes: 10 additions & 9 deletions website/docs/docs/build/validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,13 @@ These validations ensure that configuration files follow the expected schema, th

The code that handles validation [can be found here](https://github.com/dbt-labs/dbt-semantic-interfaces/tree/main/dbt_semantic_interfaces/validations) for those who want to dive deeper into this topic.


## Validations command

You can run validations from dbt Cloud or the command line with the following [MetricFlow commands](/docs/build/metricflow-commands):
You can run validations against the defined semantic model configurations from the command line with the following [MetricFlow commands](/docs/build/metricflow-commands):

Note, in dbt Cloud you don't need to validate the Semantic Layer config separately. Running a dbt command (such as `dbt parse`, `dbt build`, `dbt compile`, or `dbt run`) automatically checks it.

```bash
dbt sl validate-configs # dbt Cloud users
mf validate-configs # dbt Core users
```

Expand All @@ -32,9 +32,9 @@ In this validation step, we ensure your config files follow the defined schema f
* Dimensions
* Metrics

## Semantic
## Semantic syntax

This validation step occurs after we've built your semantic graph. The Semantic Layer, powered by MetricFlow, runs a suite of tests to ensure that your semantic graph doesn't violate any constraints. For example, we check to see if measure names are unique, or if metrics referenced in materialization exist. The current semantic rules we check for are:
This syntactic validation step occurs after we've built your semantic graph. The Semantic Layer, powered by MetricFlow, runs a suite of tests to ensure that your semantic graph doesn't violate any constraints. For example, we check to see if measure names are unique, or if metrics referenced in materialization exist. The current semantic rules we check for are:

1. Check those semantic models with measures have a valid time dimension
2. Check that there is only one primary identifier defined in each semantic model
Expand All @@ -45,9 +45,10 @@ This validation step occurs after we've built your semantic graph. The Semantic

## Data platform

This type of validation Checks to see if the semantic definitions in your semantic graph exist in the underlying physical table. To test this, we run queries against your data platform to ensure the generated SQL for semantic models, dimensions, and metrics will execute. We run the following checks
This type of validation checks to see if the semantic definitions in your semantic graph exist in the underlying physical table. To test this, we run queries against your data platform to ensure the generated SQL for semantic models, dimensions, and metrics will execute. We run the following checks:

* Check that measures and dimensions exist
* Check that underlying tables for data sources exist
* Check that the generated SQL for metrics will execute
* Measures and dimensions exist
* Underlying tables for data sources exist
* Generated SQL for metrics will execute

You can run semantic validations (against your semantic layer) in a CI job to guarantee any code changes made to dbt models don't break these metrics. For more information, refer to [semantic validation in CI](/docs/deploy/ci-jobs#semantic-validations-in-ci).
2 changes: 2 additions & 0 deletions website/docs/docs/cloud/about-cloud-develop-defer.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ pagination_next: "docs/cloud/cloud-cli-installation"

Both the dbt Cloud IDE and the dbt Cloud CLI enable users to natively defer to production metadata directly in their development workflows.

<Lightbox src src="/img/docs/reference/defer-diagram.png" width="50%" title="Use 'defer' to modify end-of-pipeline models by pointing to production models, instead of running everything upstream." />

By default, dbt follows these rules:

- dbt uses the production locations of parent models to resolve `{{ ref() }}` functions, based on metadata from the production environment.
Expand Down
8 changes: 8 additions & 0 deletions website/docs/docs/use-dbt-semantic-layer/sl-faqs.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,6 +279,14 @@ We are currently working on introducing more fine-grained access controls, inclu

## Implementation

<Expandable alt_header="How can I implement dbt Mesh with the dbt Semantic Layer">

import SLMeshFAQs from '/snippets/_sl-dbt-mesh-faq.md';

<SLMeshFAQs/>

</Expandable>

<Expandable alt_header="Which ‘staging layer’ should the dbt Semantic Layer talk to? Raw, staging, or marts?">

We recommend to build your semantic layer on top of the [marts layer](/best-practices/how-we-structure/4-marts), which represents the clean and transformed data from your dbt models.
Expand Down
1 change: 1 addition & 0 deletions website/docs/guides/mesh-qs.md
Original file line number Diff line number Diff line change
Expand Up @@ -636,6 +636,7 @@ Here are some additional resources to help you continue your journey:

- [How we build our dbt mesh projects](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro)
- [dbt Mesh FAQs](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-5-faqs)
- [Implement dbt Mesh with the Semantic Layer](/docs/use-dbt-semantic-layer/sl-faqs#how-can-i-implement-dbt-mesh-with-the-dbt-semantic-layer)
- [Cross-project references](/docs/collaborate/govern/project-dependencies#how-to-write-cross-project-ref)
- [dbt Explorer](/docs/collaborate/explore-projects)

Expand Down
2 changes: 1 addition & 1 deletion website/docs/guides/sl-snowflake-qs.md
Original file line number Diff line number Diff line change
Expand Up @@ -1056,7 +1056,7 @@ You've learned how to:
- Build, test, and manage dbt Cloud projects, focusing on metrics and semantic layers.
- Run production jobs and query metrics with our available integrations.
For next steps, you can start defining your own metrics and learn additional configuration options such as [exports](/docs/use-dbt-semantic-layer/exports), [fill null values](/docs/build/advanced-topics), and more.
For next steps, you can start defining your own metrics and learn additional configuration options such as [exports](/docs/use-dbt-semantic-layer/exports), [fill null values](/docs/build/advanced-topics), [implementing dbt Mesh with the Semantic Layer](/docs/use-dbt-semantic-layer/sl-faqs#how-can-i-implement-dbt-mesh-with-the-dbt-semantic-layer), and more.
Here are some additional resources to help you continue your journey:
Expand Down
2 changes: 2 additions & 0 deletions website/docs/reference/node-selection/defer.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ title: "Defer"

Defer is a powerful feature that makes it possible to run a subset of models or tests in a [sandbox environment](/docs/environments-in-dbt) without having to first build their upstream parents. This can save time and computational resources when you want to test a small number of models in a large project.

<Lightbox src src="/img/docs/reference/defer-diagram.png" width="50%" title="Use 'defer' to modify end-of-pipeline models by pointing to production models, instead of running everything upstream." />

Defer requires that a manifest from a previous dbt invocation be passed to the `--state` flag or env var. Together with the `state:` selection method, these features enable "Slim CI". Read more about [state](/reference/node-selection/syntax#about-node-selection).

An alternative command that accomplishes similar functionality for different use cases is `dbt clone` - see the docs for [clone](/reference/commands/clone#when-to-use-dbt-clone-instead-of-deferral) for more information.
Expand Down
30 changes: 30 additions & 0 deletions website/snippets/_sl-dbt-mesh-faq.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
When using the dbt Semantic Layer in a [dbt Mesh](/best-practices/how-we-mesh/mesh-1-intro) setting, we recommend the following:

- You have one standalone project that contains your semantic models and metrics.
- Then as you build your Semantic Layer, you can [cross-reference dbt models](/docs/collaborate/govern/project-dependencies) across your various projects or packages to create your semantic models using the [two-argument `ref` function](/reference/dbt-jinja-functions/ref#ref-project-specific-models)( `ref('project_name', 'model_name')`).
- Your dbt Semantic Layer project serves as a global source of truth across the rest of your projects.

#### Usage example
For example, let's say you have a public model (`fct_orders`) that lives in the `jaffle_finance` project. As you build your semantic model, use the following syntax to ref the model:

<File name="models/metrics/semantic_model_name.yml">

```yaml
semantic_models:
- name: customer_orders
defaults:
agg_time_dimension: first_ordered_at
description: |
Customer grain mart that aggregates customer orders.
model: ref('jaffle_finance', 'fct_orders') # ref('project_name', 'model_name')
entities:
...rest of configuration...
dimensions:
...rest of configuration...
measures:
...rest of configuration...
```
</File>
Notice that in the `model` parameter, we're using the `ref` function with two arguments to reference the public model `fct_orders` defined in the `jaffle_finance` project.
<br />
2 changes: 1 addition & 1 deletion website/snippets/cloud-feature-parity.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
The following table outlines which dbt Cloud features are supported on the different SaaS options available today. For more information about feature availability, please [contact us](https://www.getdbt.com/contact/).

| Feature | AWS Multi-tenant | AWS single tenant |Azure multi-tenant ([Preview](/docs/dbt-versions/product-lifecycles#dbt-cloud)) | Azure single tenant |
|-------------------------------|------------------|-----------------------|---------------------|------------------- -|
|-------------------------------|------------------|-----------------------|---------------------|---------------------|
| Audit logs |||||
| Continuous integration jobs |||||
| dbt Cloud CLI |||||
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit c22b03e

Please sign in to comment.