-
Notifications
You must be signed in to change notification settings - Fork 950
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add subdaily granularity #5882
add subdaily granularity #5882
Changes from 6 commits
ef816e5
6626a1c
b49a75b
c5a594d
23b373b
0c8362e
f510116
97d5125
7700ae1
e517b12
9028011
31a5fc1
0cf05df
ffb1b2e
23ac774
ce7dd8a
695a1f6
84e0c1e
eb7a262
8b71b5b
8289f21
2a35c8d
44d405f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -6,11 +6,11 @@ sidebar_label: "MetricFlow time spine" | |
tags: [Metrics, Semantic Layer] | ||
--- | ||
|
||
MetricFlow uses a timespine table to construct cumulative metrics. By default, MetricFlow expects the timespine table to be named `metricflow_time_spine` and doesn't support using a different name. | ||
MetricFlow uses a timespine table to construct cumulative metrics. By default, MetricFlow expects the timespine table to be named `metricflow_time_spine` and doesn't support using a different name. | ||
|
||
To create this table, you need to create a model in your dbt project called `metricflow_time_spine` and add the following code: | ||
To create this table, you need to create a model in your dbt project called `metricflow_time_spine` and add the following code. This example uses a `day` granularity to generate a table with one row per day. This is useful for metrics that need a daily aggregation. | ||
|
||
<File name='metricflow_time_spine.sql'> | ||
<File name='metricflow_time_spine_day.sql'> | ||
|
||
<VersionBlock lastVersion="1.6"> | ||
|
||
|
@@ -129,3 +129,40 @@ from final | |
</VersionBlock> | ||
|
||
You only need to include the `date_day` column in the table. MetricFlow can handle broader levels of detail, but it doesn't currently support finer grains. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @mirnawong1 for old versions, we can update this to say: |
||
|
||
## Hourly time spine | ||
|
||
This example uses `dbt.date_spine` with an `hour` granularity to generate a table with one row per hour. This is needed for hourly data aggregation and other sub-daily analyses. | ||
|
||
WHAT ARE OTHER OPTIONS?? TO ADD BOTH, DO USERS NEED TWO FILES (HOUR AND DAY) OR CAN THEY BE COMBINED? | ||
|
||
<File name='metricflow_time_spine_hour.sql'> | ||
|
||
```sql | ||
-- filename: metricflow_time_spine_hour.sql | ||
{{ | ||
config( | ||
materialized = 'table', | ||
) | ||
}} | ||
|
||
with hours as ( | ||
|
||
{{ | ||
dbt.date_spine( | ||
'hour', | ||
"to_date('01/01/2000','mm/dd/yyyy')", | ||
"to_date('01/01/2030','mm/dd/yyyy')" | ||
) | ||
}} | ||
|
||
), | ||
|
||
final as ( | ||
select cast(date_hour as timestamp) as date_hour | ||
from hours | ||
) | ||
|
||
select * from final | ||
``` | ||
</File> |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -9,7 +9,7 @@ pagination_next: "docs/build/cumulative" | |
|
||
Once you've created your semantic models, it's time to start adding metrics. Metrics can be defined in the same YAML files as your semantic models, or split into separate YAML files into any other subdirectories (provided that these subdirectories are also within the same dbt project repo). | ||
|
||
The keys for metrics definitions are: | ||
This page explains the different supported metric types you can add to your dbt project. The keys for metrics definitions are: | ||
matthewshaver marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
<!-- for v1.8 and higher --> | ||
|
||
|
@@ -27,6 +27,8 @@ The keys for metrics definitions are: | |
|
||
Here's a complete example of the metrics spec configuration: | ||
|
||
<File name="models/metrics/file_name.yml" > | ||
|
||
```yaml | ||
metrics: | ||
- name: metric name ## Required | ||
|
@@ -42,6 +44,8 @@ metrics: | |
{{ Dimension('entity__name') }} > 0 and {{ Dimension(' entity__another_name') }} is not | ||
null and {{ Metric('metric_name', group_by=['entity_name']) }} > 5 | ||
``` | ||
|
||
</File> | ||
</VersionBlock> | ||
|
||
<!-- for v1.7 and lower --> | ||
|
@@ -61,6 +65,8 @@ metrics: | |
|
||
Here's a complete example of the metrics spec configuration: | ||
|
||
<File name="models/metrics/file_name.yml" > | ||
|
||
```yaml | ||
metrics: | ||
- name: metric name ## Required | ||
|
@@ -76,19 +82,26 @@ metrics: | |
{{ Dimension('entity__name') }} > 0 and {{ Dimension(' entity__another_name') }} is not | ||
null and {{ Metric('metric_name', group_by=['entity_name']) }} > 5 | ||
``` | ||
</File> | ||
|
||
</VersionBlock> | ||
|
||
This page explains the different supported metric types you can add to your dbt project. | ||
|
||
import SLCourses from '/snippets/_sl-course.md'; | ||
|
||
<SLCourses/> | ||
|
||
### Conversion metrics | ||
## Sub-daily granularity | ||
|
||
Sub-daily granularity enables you to query metrics at granularities at finer time grains below a day, such as hourly, minute, or even by the second. It support anything that `date_trunc` supports. This can be useful if you want more detailed analysis and for datasets where you need more granular time data, such as minute-by-minute event tracking. | ||
|
||
For more configuration details, refer to [sub-daily granularity](/docs/build/granularity). | ||
|
||
## Conversion metrics | ||
|
||
[Conversion metrics](/docs/build/conversion) help you track when a base event and a subsequent conversion event occur for an entity within a set time period. | ||
|
||
<File name="models/metrics/file_name.yml" > | ||
|
||
```yaml | ||
metrics: | ||
- name: The metric name | ||
|
@@ -112,11 +125,14 @@ metrics: | |
- base_property: DIMENSION or ENTITY | ||
conversion_property: DIMENSION or ENTITY | ||
``` | ||
</File> | ||
|
||
### Cumulative metrics | ||
## Cumulative metrics | ||
|
||
[Cumulative metrics](/docs/build/cumulative) aggregate a measure over a given window. If no window is specified, the window will accumulate the measure over all of the recorded time period. Note that you will need to create the [time spine model](/docs/build/metricflow-time-spine) before you add cumulative metrics. | ||
|
||
<File name="models/metrics/file_name.yml" > | ||
|
||
```yaml | ||
# Cumulative metrics aggregate a measure over a given window. The window is considered infinite if no window parameter is passed (accumulate the measure over all of time) | ||
metrics: | ||
|
@@ -130,11 +146,14 @@ metrics: | |
join_to_timespine: true | ||
window: 7 days | ||
``` | ||
</File> | ||
|
||
### Derived metrics | ||
## Derived metrics | ||
|
||
[Derived metrics](/docs/build/derived) are defined as an expression of other metrics. Derived metrics allow you to do calculations on top of metrics. | ||
|
||
<File name="models/metrics/file_name.yml" > | ||
|
||
```yaml | ||
metrics: | ||
- name: order_gross_profit | ||
|
@@ -149,6 +168,8 @@ metrics: | |
- name: order_cost | ||
alias: cost | ||
``` | ||
</File> | ||
|
||
<!-- not supported | ||
### Expression metrics | ||
Use [expression metrics](/docs/build/expr) when you're building a metric that involves a SQL expression of multiple measures. | ||
|
@@ -167,10 +188,12 @@ metrics: | |
``` | ||
--> | ||
|
||
### Ratio metrics | ||
## Ratio metrics | ||
|
||
[Ratio metrics](/docs/build/ratio) involve a numerator metric and a denominator metric. A `filter` string can be applied to both the numerator and denominator or separately to the numerator or denominator. | ||
|
||
<File name="models/metrics/file_name.yml" > | ||
|
||
```yaml | ||
metrics: | ||
- name: cancellation_rate | ||
|
@@ -191,15 +214,18 @@ metrics: | |
filter: | | ||
{{ Dimension('customer__country') }} = 'MX' | ||
``` | ||
</File> | ||
|
||
### Simple metrics | ||
## Simple metrics | ||
|
||
[Simple metrics](/docs/build/simple) point directly to a measure. You may think of it as a function that takes only one measure as the input. | ||
|
||
- `name` — Use this parameter to define the reference name of the metric. The name must be unique amongst metrics and can include lowercase letters, numbers, and underscores. You can use this name to call the metric from the dbt Semantic Layer API. | ||
|
||
**Note:** If you've already defined the measure using the `create_metric: True` parameter, you don't need to create simple metrics. However, if you would like to include a constraint on top of the measure, you will need to create a simple type metric. | ||
|
||
<File name="models/metrics/file_name.yml" > | ||
|
||
```yaml | ||
metrics: | ||
- name: cancellations | ||
|
@@ -214,13 +240,16 @@ metrics: | |
{{ Dimension('order__value')}} > 100 and {{Dimension('user__acquisition')}} is not null | ||
join_to_timespine: true | ||
``` | ||
</File> | ||
|
||
## Filters | ||
|
||
A filter is configured using Jinja templating. Use the following syntax to reference entities, dimensions, time dimensions, or metrics in filters. | ||
|
||
Refer to [Metrics as dimensions](/docs/build/ref-metrics-in-filters) for details on how to use metrics as dimensions with metric filters: | ||
|
||
<File name="models/metrics/file_name.yml" > | ||
|
||
```yaml | ||
filter: | | ||
{{ Entity('entity_name') }} | ||
|
@@ -232,10 +261,19 @@ filter: | | |
{{ TimeDimension('time_dimension', 'granularity') }} | ||
|
||
filter: | | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @mirnawong1 Can we version block this metric filter example to versions 1.8+? |
||
{{ Metric('metric_name', group_by=['entity_name']) }} # Available in v1.8 or go versionless with [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) | ||
{{ Metric('metric_name', group_by=['entity_name']) }} {# Available in v1.8 or go [versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version). } | ||
``` | ||
|
||
</File> | ||
|
||
For example, if I wanted to filter for the order date dimension, grouped by month, I'd use the following syntax: | ||
matthewshaver marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
```yaml | ||
filter: | | ||
{{ TimeDimension('order_date', 'month') }} | ||
``` | ||
|
||
### Further configuration | ||
## Further configuration | ||
|
||
You can set more metadata for your metrics, which can be used by other tools later on. The way this metadata is used will vary based on the specific integration partner | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,64 @@ | ||
--- | ||
title: Sub-daily granularity | ||
id: "granularity" | ||
description: "Sub-daily granularity enables you to query metrics at granularities at finer time grains below a day, such as hourly, minute, or even by the second. " | ||
sidebar_label: "Sub-daily granularity" | ||
tags: [Metrics, Semantic Layer] | ||
pagination_next: "docs/build/conversion" | ||
--- | ||
|
||
|
||
Sub-daily granularity enables you to query metrics at granularities at finer time grains such as hourly, minute, or even by the second. It support anything that `date_trunc` supports. You can sub-daily granularity for cumulative metrics, time spine models at sub-daily grains, or default grain settings for metrics. | ||
|
||
MetricFlow defaults to the `day` grain, while allowing you the ability to default granularity as a metric-level property. | ||
|
||
This is particularly useful for more detailed analysis and for datasets where high-resolution time data is required, such as minute-by-minute event tracking. | ||
|
||
### Usage | ||
There are two ways to specify sub-daily granularity: `default_grain` and `time_granularity`. This section explains how to use both methods, how they also interact, and which one takes precedence. | ||
|
||
- #### `default_grain` | ||
Use the `default_grain` parameter in the metric-level metric config to specify the default granularity for querying the metric when no specific granularity is defined. It allows specifying the most common or sensible default, like day, hour, and so on. | ||
|
||
This parameter is optional and defaults to `day`. | ||
|
||
<File name="models/metrics/file_name.yml" > | ||
|
||
```yaml | ||
metrics: | ||
- name: my_metrics | ||
... | ||
default_grain: day # Optional: defaults to day | ||
``` | ||
</File> | ||
|
||
- #### `time_granularity` | ||
Use the `time_granularity` parameter at the dimension-level with the [time dimension](/docs/build/dimensions#time) `type_params` to specify the level of granularity directly on the data, like hour, minute, second, and so on. It affects how the data is truncated or aggregated in queries. | ||
|
||
<File name="models/metrics/file_name.yml" > | ||
|
||
```yaml | ||
dimensions: | ||
- name: ordered_at | ||
type: time | ||
type_params: | ||
time_granularity: hour | ||
|
||
``` | ||
</File> | ||
|
||
### Precedence | ||
When querying metrics by `metric_time`, MetricFlow currently defaults to the grain of the `agg_time_dimension`. If you want to query metrics at a different grain, you can use the `time_granularity` type parameter in time dimensions. | ||
|
||
The following table explains how `default_grain` and `time_granularity` interact and the resulting query granularity: | ||
|
||
| Context | `default_grain` | `time_granularity` | Result | | ||
| --- | --- | --- | --- | | ||
| Only `default_grain` specified | `day` | `hour` | Query at `hour` granularity | | ||
| Only `time_granularity` specified | - | `hour` | Query at `hour` granularity | | ||
| Both specified, same value | `hour` | `hour` | Query at `hour` granularity | | ||
| Both specified, different value | `day` | `minute` | Query at `minute` granularity | | ||
| Both not specified | - | - | Defaults to `day` | | ||
|
||
Implementation using the `time_granularity` type parameter in time dimensions. | ||
Examples of using DATE_TRUNC with sub-daily granularities in SQL. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mirnawong1 For this entire file - can we use this deleted text for versions <=1.8, instead of the new text? The new text should only be for 1.9+.