Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release Normalize version 0.4.0 #1081

Merged
merged 5 commits into from
Dec 4, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -41,11 +41,12 @@ import {versions} from '@site/src/componentVersions';
<TabItem value="normalize" label="Snowplow Normalize">

<ReactMarkdown children={`
| snowplow-normalize version | dbt versions | BigQuery | Databricks | Redshift | Snowflake | Postgres |
| -------------------------------- | ----------------- | :------: | :--------: | :------: | :-------: | :------: |
| ${versions.dbtSnowplowNormalize} | >=1.4.0 to <2.0.0 | ✅ | ✅ | ❌ | ✅ | ❌ |
| 0.2.3 | >=1.3.0 to <2.0.0 | ✅ | ✅ | ❌ | ✅ | ❌ |
| 0.1.0 | >=1.0.0 to <2.0.0 | ✅ | ✅ | ❌ | ✅ | ❌ |
| snowplow-normalize version | dbt versions | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| -------------------------------- | ----------------- | :------: | :--------: | :------: | :-------: | :------: | :------: |
| ${versions.dbtSnowplowNormalize} | >=1.4.0 to <2.0.0 | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ |
| 0.3.5 | >=1.4.0 to <2.0.0 | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ |
| 0.2.3 | >=1.3.0 to <2.0.0 | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ |
| 0.1.0 | >=1.0.0 to <2.0.0 | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ |
`} remarkPlugins={[remarkGfm]} />

</TabItem>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,44 +24,12 @@ Normalize in this context means [database normalization](https://en.wikipedia.or

**The package source code can be found in the [snowplow/dbt-snowplow-normalize repo](https://github.com/snowplow/dbt-snowplow-normalize), and the docs for the [macro design are here](https://snowplow.github.io/dbt-snowplow-normalize/#/overview/snowplow_normalize).**

## Package Configuration

### Partition Timestamp Configuration

The package uses a configurable partition timestamp column, controlled by the `snowplow__partition_tstamp` variable:

```yaml
vars:
snowplow__partition_tstamp: "collector_tstamp" # Default value
```

:::warning Important Note on Custom Partition Timestamps
If you change `snowplow__partition_tstamp` to a different column (e.g., "loader_tstamp"), you MUST ensure that this column is included in the `event_columns` list in your normalize configuration for each event. Failing to do so will cause the models to fail, as the partition column must be present in the normalized output.

Example configuration when using a custom partition timestamp:
```json
{
"events": [
{
"event_names": ["page_view"],
"event_columns": [
"domain_userid",
"loader_tstamp", // Must include your custom partition timestamp here
"app_id"
],
// ... rest of configuration
}
]
}
```
:::

The package provides [macros](https://docs.getdbt.com/docs/build/jinja-macros) and a python script that is used to generate your normalized events, filtered events, and users table for use within downstream ETL tools such as Census. See the [Model Design](#model-design) section for further details on these tables.

The package only includes the base incremental scratch model and does not have any derived models, instead it generates models in your project as if they were custom models you had built on top of the [Snowplow incremental tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/index.md), using the `_this_run` table as the base for new events to process each run. See the [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/index.md) section for the variables that apply to the incremental model.

:::note
The incremental model is simplified compared to the standard unified model, this package does not use sessions to identify which historic events to reprocess and just uses the `snowplow__partition_tstamp` (defaults to `collector_tstamp`) and package variables to identify which events to (re)process.
The incremental model is simplified compared to the standard unified model, this package does not use sessions to identify which historic events to reprocess and just uses the `snowplow__session_timestamp` (defaults to `collector_tstamp`) and package variables to identify which events to (re)process.
:::

## Model Design
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -114,8 +114,37 @@ Depending on the use case it should either be the catalog (for Unity Catalog use

:::

### 10. Partition Timestamp Configuration
ilias1111 marked this conversation as resolved.
Show resolved Hide resolved

### 10. Run your model(s)
The package uses a configurable partition timestamp column, controlled by the `snowplow__partition_tstamp` variable:

```yaml
vars:
snowplow__partition_tstamp: "collector_tstamp" # Default value
```
agnessnowplow marked this conversation as resolved.
Show resolved Hide resolved

:::warning Important Note on Custom Partition Timestamps
If you change `snowplow__partition_tstamp` to a different column (e.g., "loader_tstamp"), you MUST ensure that this column is included in the `event_columns` list in your normalize configuration for each event. Failing to do so will cause the models to fail, as the partition column must be present in the normalized output.

Example configuration when using a custom partition timestamp:
```json
{
"events": [
{
"event_names": ["page_view"],
"event_columns": [
"domain_userid",
"loader_tstamp", // Must include your custom partition timestamp here
"app_id"
],
// ... rest of configuration
}
]
}
```
:::

### 11. Run your model(s)

You can now run your models for the first time by running the below command (see the [operation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/index.md) page for more information on operation of the package):

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@
"description": "Determines which timestamp is used to process sessions of data"
},
"snowplow__partition_tstamp": {
"recommendFullRefresh": false,
"recommendFullRefresh": true,
"order": 56,
"consoleGroup": "advanced",
"type": "string",
Expand Down