Skip to content

Commit

Permalink
add documentation for unit testing incremental models (#4959)
Browse files Browse the repository at this point in the history
## What are you changing in this pull request and why?
<!---
Describe your changes and why you're making them. If related to an open 
issue or a pull request on dbt Core, then link to them here! 

To learn more about the writing conventions used in the dbt Labs docs,
see the [Content style
guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md).
-->

Closes #4929

## Checklist
<!--
Uncomment when publishing docs for a prerelease version of dbt:
- [ ] Add versioning components, as described in [Versioning
Docs](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-entire-pages)
- [ ] Add a note to the prerelease version [Migration
Guide](https://github.com/dbt-labs/docs.getdbt.com/tree/current/website/docs/docs/dbt-versions/core-upgrade)
-->
- [ ] Review the [Content style
guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md)
so my content adheres to these guidelines.
- [ ] For [docs
versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#about-versioning),
review how to [version a whole
page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version)
and [version a block of
content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content).
- [ ] Add a checklist item for anything that needs to happen before this
PR is merged, such as "needs technical review" or "change base branch."

Adding or removing pages (delete if not applicable):
- [ ] Add/remove page in `website/sidebars.js`
- [ ] Provide a unique filename for new pages
- [ ] Add an entry for deleted pages in `website/static/_redirects`
- [ ] Run link testing locally with `npm run build` to update the links
that point to deleted pages
  • Loading branch information
graciegoheen authored Feb 22, 2024
2 parents 2f7af3b + 2b40e65 commit 97effa2
Show file tree
Hide file tree
Showing 2 changed files with 73 additions and 0 deletions.
69 changes: 69 additions & 0 deletions website/docs/docs/build/unit-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,5 +255,74 @@ unit_tests:

```

## Unit testing incremental models

When configuring your unit test, you can override the output of macros, vars, or environment variables. This enables you to unit test your incremental models in "full refresh" and "incremental" modes.

When testing an incremental model, the expected output is the __result of the materialization__ (what will be merged/inserted), not the resulting model itself (what the final table will look like after the merge/insert).

For example, say you have an incremental model in your project:

<File name='my_incremental_model.sql'>

```sql

{{
config(
materialized='incremental'
)
}}

select * from {{ ref('events') }}
{% if is_incremental() %}
where event_time > (select max(event_time) from {{ this }})
{% endif %}

```

</File>

You can define unit tests on `my_incremental_model` to ensure your incremental logic is working as expected:

```yml

unit_tests:
- name: my_incremental_model_full_refresh_mode
model: my_incremental_model
overrides:
macros:
# unit test this model in "full refresh" mode
is_incremental: false
given:
- input: ref('events')
rows:
- {event_id: 1, event_time: 2020-01-01}
expect:
rows:
- {event_id: 1, event_time: 2020-01-01}

- name: my_incremental_model_incremental_mode
model: my_incremental_model
overrides:
macros:
# unit test this model in "incremental" mode
is_incremental: true
given:
- input: ref('events')
rows:
- {event_id: 1, event_time: 2020-01-01}
- {event_id: 2, event_time: 2020-01-02}
- {event_id: 3, event_time: 2020-01-03}
- input: this
# contents of current my_incremental_model
rows:
- {event_id: 1, event_time: 2020-01-01}
expect:
# what will be inserted/merged into my_incremental_model
rows:
- {event_id: 2, event_time: 2020-01-02}
- {event_id: 3, event_time: 2020-01-03}

```

There is currently no way to unit test whether the dbt framework inserted/merged the records into your existing model correctly, but [we're investigating support for this in the future](https://github.com/dbt-labs/dbt-core/issues/8664).
4 changes: 4 additions & 0 deletions website/docs/reference/resource-properties/unit-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,10 @@ When `format: csv`, can either supply:
- If you do supply an input for a seed, we will use that input instead.
- You can also have “empty” inputs, by setting rows to an empty list `rows: []`

### Overrides

When configuring your unit test, you can override the output of macros, vars, or environment variables for a given unit test.

## Examples
```yml
Expand Down

0 comments on commit 97effa2

Please sign in to comment.