Skip to content

Commit

Permalink
Merge branch 'current' into patch-1
Browse files Browse the repository at this point in the history
  • Loading branch information
mirnawong1 authored Nov 30, 2022
2 parents dcac3d9 + 38cc18a commit 1ec3e12
Show file tree
Hide file tree
Showing 216 changed files with 9,772 additions and 4,787 deletions.
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,14 @@ You can use components documented in the [docusaurus library](https://v2.docusau

# Writing content

When writing content, you should refer to the [style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) and [content types](/contributing/content-types.md) to help you understand our writing standards and how we break down information in the product documentation.
The dbt Labs docs are written in Markdown and sometimes HTML. When writing content, refer to the [style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) and [content types](/contributing/content-types.md) to help you understand our writing standards and how we break down information in the product documentation.

## SME and editorial reviews

All PRs that are submitted will be reviewed by the dbt Labs Docs team for editorial review.

Content that is submitted by our users and the open-source community are also reviewed by our dbt Labs subject matter experts (SMEs) to help ensure technical accuracy.


## Versioning and single-sourcing content

Expand Down
8 changes: 8 additions & 0 deletions _redirects
Original file line number Diff line number Diff line change
Expand Up @@ -448,6 +448,8 @@ https://tutorial.getdbt.com/* https://docs.getdbt.com/:splat 301!
docs/contributing/building-a-new-adapter /docs/contributing/adapter-development/3-building-a-new-adapter 302
docs/contributing/testing-a-new-adapter /docs/contributing/adapter-development/4-testing-a-new-adapter 302
docs/contributing/documenting-a-new-adapter /docs/contributing/adapter-development/5-documenting-a-new-adapter 302

/docs/dbt-cloud/using-dbt-cloud/cloud-metrics-layer /docs/use-dbt-semantic-layer/dbt-semantic-layer 301!
/docs/building-a-new-adapter /docs/contributing/adapter-development/3-building-a-new-adapter 301!
/reference/warehouse-profiles/impala-profile /reference/warehouse-setups/impala-setup 302
/reference/warehouse-profiles/exasol-profile /reference/warehouse-setups/exasol-setup 302
Expand Down Expand Up @@ -482,6 +484,7 @@ docs/contributing/documenting-a-new-adapter /docs/contributing/adapter-developme
/reference/warehouse-profiles/firebolt-profile /reference/warehouse-setups/firebolt-setup 302
/reference/warehouse-profiles/mysql-profile /reference/warehouse-setups/mysql-setup 302
/reference/warehouse-profiles/hive-profile /reference/warehouse-setups/hive-setup 302
/reference/using-sources /docs/build/sources 302

# getting started guide
/guides/getting-started /docs/get-started/getting-started/overview 301
Expand All @@ -500,6 +503,7 @@ docs/contributing/documenting-a-new-adapter /docs/contributing/adapter-developme
/guides/getting-started/learning-more/getting-started-dbt-core /docs/get-started/getting-started-dbt-core 301
/guides/getting-started/learning-more/refactoring-legacy-sql /docs/get-started/learning-more/refactoring-legacy-sql 301
/guides/getting-started/learning-more/using-jinja /docs/get-started/learning-more/using-jinja 301

# ide ia redirects
/docs/dbt-cloud/cloud-ide/the-dbt-ide /docs/getting-started/dbt-cloud-features 301!
/docs/dbt-cloud/cloud-ide/handling-merge-conflicts /docs/collaborate/git/resolve-merge-conflicts 301!
Expand All @@ -517,3 +521,7 @@ docs/contributing/documenting-a-new-adapter /docs/contributing/adapter-developme
/community/maintaining-a-channel /community/resources/maintaining-a-channel 301
/docs/contributing/oss-expectations /community/resources/oss-expectations 301
/docs/contributing/slack-rules-of-the-road /community/resources/slack-rules-of-the-road 301

/blog/getting-started-with-the-dbt-semantic-layer /blog/understanding-the-components-of-the-dbt-semantic-layer 301!
/docs/getting-started/develop-in-the-cloud#creating-a-development-environment /docs/get-started/develop-in-the-cloud#set-up-and-access-the-cloud-ide 301

2 changes: 1 addition & 1 deletion contributing/content-types.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ Procedural content should include troubleshooting tips as frequently as possible

## Guide

Guides (formerly called long-form procedural articles) are highly-approachable articles that group information in context to help readers complete a complex task or set of related tasks. Guides eliminate duplication and ensure the customer finds contextual content in the right place. Guides may be a set of tasks within the reader’s larger workflow, such as including use cases.
Guides are highly-approachable articles that group information in context to help readers complete a complex task or set of related tasks. Guides eliminate duplication and ensure people find contextual content in the right place. Guides may be a set of tasks within the reader’s larger workflow, such as including use cases.

Guides combine the content types within a single article to illustrate an entire workflow within a single page, rather than splitting the workflow out into separate pieces. Guides containing multiple procedures help us scale as more options are added to the product. Users may need to complete different procedures within the guide at different times, or refer back to the guide for conceptual content or to complete a followup task.
Example usage: If there is a large number of the same type of setting, use a guide that gathers all of the tasks in context.
Expand Down
6 changes: 6 additions & 0 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

22 changes: 10 additions & 12 deletions website/blog/2019-05-01-how-we-structure-dbt-projects.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,6 @@ In comparison, the (recently updated) [best practices](/guides/best-practices) r
* Consider the information architecture of your data warehouse
* Separate source-centric and business-centric transformations

If you want to see what the code for one of our projects looks like, check out [this demonstration dbt project](https://github.com/dbt-labs/dbt-learn-demo/tree/day2-dbt-training/models).

We also recently held (and recorded) an office hours on this topic – this article provides a high level outline, but there’s a lot more detail and discussion in the [video](https://youtu.be/xzKLh342s08).

Lastly, before I dive in, a huge thank you to Jeremy Cohen for not only teaching me a lot of the material in this article, but also for doing a lot of the groundwork that went into this article – entire sections of this article are in fact lifted from his work.
Expand Down Expand Up @@ -69,7 +67,7 @@ In our dbt projects, this leads us to our first split in our `models/` directory
└── models
├── marts
└── staging
```

## Staging raw data
Expand Down Expand Up @@ -119,7 +117,7 @@ Each staging directory contains at a minimum:
├── stg_braintree__customers.sql
└── stg_braintree__payments.sql
```


Some dbt users prefer to have one `.yml` file per model (e.g. `stg_braintree__customers.yml`). This is a completely reasonable choice, and we recommend implementing it if your `.yml` files start to become unwieldy.

Expand All @@ -133,23 +131,23 @@ That being said, in our dbt projects every source flows through exactly one mode

```
with source as (
select * from {{ source('braintree', 'payments') }}
),
renamed as (
select
id as payment_id,
order_id,
convert_timezone('America/New_York', 'UTC', createdat) as created_at,
...
from source
)
select * from renamed
```

Expand All @@ -175,7 +173,7 @@ In our dbt projects, we place these base models in a nested `base` subdirectory.
├── stg_braintree.yml
├── stg_braintree__customers.sql
└── stg_braintree__payments.sql
```
```

In our projects, base models:

Expand Down
40 changes: 20 additions & 20 deletions website/blog/2021-02-05-dbt-project-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "Your Essential dbt Project Checklist"
description: "A checklist created to guide our internal work, which you can use to clean up your own dbt project."
slug: essential-dbt-project-checklist
canonical_url: https://discourse.getdbt.com/t/your-essential-dbt-project-checklist/1377
canonical_url: https://discourse.getdbt.com/t/your-essential-dbt-project-checklist/1377

authors: [amy_chen, dave_connors]

Expand All @@ -15,7 +15,7 @@ is_featured: true

If you’ve been using dbt for over a year, your project is out-of-date. This is natural.

New functionalities have been released. Warehouses change. Best practices are updated. Over the last year, I and others on the Fishtown Analytics (now dbt Labs!) team have conducted seven audits for clients who have been using dbt for a minimum of 2 months.
New functionalities have been released. Warehouses change. Best practices are updated. Over the last year, I and others on the Fishtown Analytics (now dbt Labs!) team have conducted seven audits for clients who have been using dbt for a minimum of 2 months.

<!--truncate-->

Expand Down Expand Up @@ -119,7 +119,7 @@ This post is the checklist I created to guide our internal work, and I’m shari
* Do you use refs and sources for everything?
* Make sure nothing is querying off of raw tables, etc.
![no querying raw tables](/img/blog/checklist-8ddc2f76de24c98690ef986dcc7974bff09adb59.png)

* Do you regularly run `dbt test` as part of your workflow and production jobs?
* Do you use Jinja & Macros for repeated code?
* If you do, is the balance met where it’s not being overused to the point code is not readable?
Expand Down Expand Up @@ -200,7 +200,7 @@ Are you using the IDE and if so, how well?

**Useful links**

* [dbt Cloud as a CI tool](/docs/dbt-cloud/using-dbt-cloud/cloud-enabling-continuous-integration-with-github)
* [dbt Cloud as a CI tool](/docs/deploy/cloud-ci-job)


## ✅ DAG Auditing
Expand All @@ -210,41 +210,41 @@ _Note: diagrams in this section show what NOT to do!_

* Does your DAG have any common modeling pitfalls?
* Are there any direct joins from sources into an intermediate model?

* All sources should have a corresponding staging model to clean and standardize the data structure. They should not look like the image below.

![bad dag](/img/blog/checklist-28c75101367e272fbc2db2ebb1a1ec030517bb5e_2_517x250.jpeg)

* Do sources join directly together?

* All sources should have a corresponding staging model to clean and standardize the data structure. They should not look like the image below.

![bad dag 2](/img/blog/checklist-5d8ad45deb695eb6771003e010b242c0a3c122b9_2_517x220.jpeg)

* Are there any rejoining of upstream concepts?

* This may indicate:
* a model may need to be expanded so all the necessary data is available downstream
* a new intermediate model is necessary to join the concepts for use in both places

![bad dag 2](/img/blog/checklist-acd57c0e781b1eaf75a65b5063f97ac3ddc5c493_2_517x136.jpeg)

* Are there any “bending connections”?

* Are models in the same layer dependent on each other?
* This may indicate a change in naming is necessary, or the model should reference further upstream models

![bad dag 3](/img/blog/checklist-0532fd13a7d63e3e5df71d025700c4d9c158a7ff_2_517x155.jpeg)

* Are there model fan outs of intermediate/dimension/fact models?

* This might indicate some transformations should move to the BI layer, or transformations should be moved upstream
* Your dbt project needs a defined end point!

[![bad dag 4](/img/blog/checklist-33fcd7c4922233412d1364b39227c876d0cb8215_2_517x111.jpeg)

* Is there repeated logic found in multiple models?

* This indicates an opportunity to move logic into upstream models or create specific intermediate models to make that logic reusable
* One common place to look for this is complex join logic. For example, if you’re checking multiple fields for certain specific values in a join, these can likely be condensed into a single field in an upstream model to create a clean, simple join.

Expand Down
11 changes: 5 additions & 6 deletions website/blog/2021-09-11-sql-dateadd.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ The *functions themselves* are named slightly differently, which is common acros

```sql
dateadd( {{ datepart }}, {{ interval }}, {{ from_date }} )
```
```

*Hour, minute and second are supported!*

Expand All @@ -57,14 +57,14 @@ date_add( {{ startDate }}, {{ numDays }} )

```sql
date_add( {{ from_date }}, INTERVAL {{ interval }} {{ datepart }} )
```
```

*Dateparts of less than a day (hour / minute / second) are not supported.*

### The DATEADD Function in Postgres...

Postgres doesn’t provide a dateadd function out of the box, so you’ve got to go it alone - but the syntax looks very similar to BigQuery’s function…

```sql
{{ from_date }} + (interval '{{ interval }} {{ datepart }}')
```
Expand Down Expand Up @@ -96,7 +96,7 @@ Adding 1 month to today would look like...
```

> *New to dbt? Check out [dbt introduction](https://docs.getdbt.com/docs/introduction) for more background on dbt and the analytics engineering workflow that it facilitates.*
>
>
> *TL;DR: dbt allows data practitioners to write code like software engineers, which in this case means not repeating yourself unnecessarily.*
### Compiling away your DATEADD troubles
Expand Down Expand Up @@ -143,5 +143,4 @@ Enjoy! FYI I've used dateadd macro in dbt-utils on BigQuery, Postgres, Redshift

*Note: While `dbt_utils` doesn't support Databricks by default, you can use other packages that [implement overrides](/reference/dbt-jinja-functions/dispatch#overriding-package-macros) as a workaround.*

*This [spark_utils package](https://github.com/dbt-labs/spark-utils/blob/main/macros/dbt_utils/cross_db_utils/dateadd.sql) can help you implement the override needed to add support for Databricks dateadd*

*This [spark_utils package](https://github.com/dbt-labs/spark-utils/blob/0.3.0/macros/dbt_utils/cross_db_utils/dateadd.sql) can help you implement the override needed to add support for Databricks dateadd*
8 changes: 4 additions & 4 deletions website/blog/2021-10-15-october-21-product-update-email.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "October 2021 dbt Update: Metrics and Hat Tricks 🎩"
description: "Also flagging that Coalesce is less than 3 weeks away! 😱"
slug: dbt-product-update-2021-october
authors: [lauren_craigie]
authors: [lauren_craigie]

tags: [dbt updates]
hide_table_of_contents: false
Expand Down Expand Up @@ -53,7 +53,7 @@ I've got three really exciting things to share this month!

- [Model bottlenecks beta](https://getdbt.slack.com/archives/C02GUTGK73N?utm_campaign=Monthly%20Product%20Updates&utm_source=hs_email&utm_medium=email&_hsenc=p2ANqtz-_wfy8vfjMjwQ7o8TXEOVz-oXI35iVcVP1HtAvriVHfJoAd1IcsP-MCww6vJUDlvAfiuQjZ): Identify long-running models ripe for refactoring (or re-scheduling). The new model timing dashboard in the run detail page helps you quickly assess job composition, order, and duration to optimize your workflows and cut costs💰

![image-1](https://hs-8698602.f.hubspotemail.net/hub/8698602/hubfs/image-1.png?upscale=true&width=1120&upscale=true&name=image-1.png)
![image-1](https://hs-8698602.f.hubspotemail.net/hub/8698602/hubfs/image-1.png?upscale=true&width=1120&upscale=true&name=image-1.png)

The Model Timing tab in dbt Cloud highlights models taking particularly long to run.

Expand All @@ -78,10 +78,10 @@ The Model Timing tab in dbt Cloud highlights models taking particularly long to
At the Future Data Conference last week Tristan noted that data workflows borrow much from software engineering, but haven't really crossed the DevOps chasm. What's missing? Spreadsheets? Actually... *maybe.* 😅 Okay you had to be there. Luckily you still can! Check out the [recording](https://futuredata.brighttalk.live/talk/19069-506932/?utm_campaign=Monthly%20Product%20Updates&utm_source=hs_email&utm_medium=email&_hsenc=p2ANqtz-_wfy8vfjMjwQ7o8TXEOVz-oXI35iVcVP1HtAvriVHfJoAd1IcsP-MCww6vJUDlvAfiuQjZ).
- [Modeling behavioral data with Snowplow and dbt](https://get.snowplowanalytics.com/wbn/dbt-and-snowplow/data-modeling/?utm_campaign=Monthly%20Product%20Updates&utm_source=hs_email&utm_medium=email&_hsenc=p2ANqtz-_wfy8vfjMjwQ7o8TXEOVz-oXI35iVcVP1HtAvriVHfJoAd1IcsP-MCww6vJUDlvAfiuQjZ) (coming up on 10/27).
Our own Sanjana Sen joins the Snowplow team to talk modeling Snowplow event data in dbt -- including how to structure your data models, best practices to follow, and key pitfalls to avoid.
- [How Blend Eliminated Data Silos with dbt and Hightouch](https://hightouch.io/dbt-hightouch-blend-event/?utm_campaign=Monthly%20Product%20Updates&utm_source=hs_email&utm_medium=email&_hsenc=p2ANqtz-_wfy8vfjMjwQ7o8TXEOVz-oXI35iVcVP1HtAvriVHfJoAd1IcsP-MCww6vJUDlvAfiuQjZ) (coming up 10/28).
- How Blend Eliminated Data Silos with dbt and Hightouch.
Fin-tech behemoth, Blend, processes trillions of dollars in loans (and recently IPO'd). Join this talk with William Tsu (Customer Success Operations at Blend) to learn how adopting dbt and Hightouch has helped them overcome data silos to keep kicking a$$.


That's all for now! Thanks for reading, and as always, *let me know if there's anything else you want to see in these updates!*

*Lauren Craigie*
Expand Down
2 changes: 1 addition & 1 deletion website/blog/2021-11-23-so-you-want-to-build-a-package.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ If you’re considering making a package, you probably already know what one is

Packages are a way to share code in dbt without ever having to copy and paste (or *email* :screaming face:).

Let’s break down the [dateadd macro](https://github.com/dbt-labs/dbt-utils/blob/main/macros/cross_db_utils/dateadd.sql) from the dbt_utils macro to show you the process that created this fantastic macro.
Let’s break down the [dateadd macro](https://github.com/dbt-labs/dbt-utils/blob/0.1.20/macros/cross_db_utils/dateadd.sql) from the dbt_utils macro to show you the process that created this fantastic macro.

The problem: Analysts often need to add an interval to a timestamp/date. To make this cross-database and standardized across a project, a macro is needed.

Expand Down
Loading

0 comments on commit 1ec3e12

Please sign in to comment.