-
Notifications
You must be signed in to change notification settings - Fork 948
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
update sl ide support #6072
update sl ide support #6072
Changes from 4 commits
ab2202f
a7cd168
a0e534b
c55ddb0
1132632
bf556a4
dfae29e
1e377aa
d14d4f5
e809426
8a44f77
24fcd72
b30af36
fede479
8980899
0083d31
dc334b9
fffc37f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -8,7 +8,7 @@ tags: [Metrics, Semantic Layer] | |
|
||
Once you define metrics in your dbt project, you can query metrics, dimensions, and dimension values, and validate your configs using the MetricFlow commands. | ||
|
||
MetricFlow allows you to define and query metrics in your dbt project in the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation), [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud), or [dbt Core](/docs/core/installation-overview). To experience the power of the universal [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) and dynamically query those metrics in downstream tools, you'll need a dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) account. | ||
MetricFlow allows you to define and query metrics in your dbt project in the [dbt Cloud](/docs/cloud/about-develop-dbt) or [dbt Core](/docs/core/installation-overview). To experience the power of the universal [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) and dynamically query those metrics in downstream tools, you'll need a dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) account. | ||
|
||
MetricFlow is compatible with Python versions 3.8, 3.9, 3.10, and 3.11. | ||
|
||
|
@@ -18,33 +18,18 @@ MetricFlow is a dbt package that allows you to define and query metrics in your | |
|
||
Using MetricFlow with dbt Cloud means you won't need to manage versioning — your dbt Cloud account will automatically manage the versioning. | ||
|
||
**dbt Cloud jobs** — MetricFlow commands aren't supported in dbt Cloud jobs yet. However, you can add MetricFlow validations with your git provider (such as GitHub Actions) by installing MetricFlow (`python -m pip install metricflow`). This allows you to run MetricFlow commands as part of your continuous integration checks on PRs. | ||
dbt Cloud jobs support the `dbt sl validate` command to [automatically test your semantic nodes](/docs/deploy/ci-jobs#semantic-validations-in-ci). You can also add MetricFlow validations with your git provider (such as GitHub Actions) by installing MetricFlow (`python -m pip install metricflow`). This allows you to run MetricFlow commands as part of your continuous integration checks on PRs. | ||
|
||
<Tabs> | ||
|
||
<TabItem value="cloudcli" label="dbt Cloud CLI"> | ||
<TabItem value="cloud" label="MetricFlow with dbt Cloud"> | ||
|
||
- MetricFlow [commands](#metricflow-commands) are embedded in the dbt Cloud CLI. This means you can immediately run them once you install the dbt Cloud CLI and don't need to install MetricFlow separately. | ||
- You don't need to manage versioning — your dbt Cloud account will automatically manage the versioning for you. | ||
|
||
</TabItem> | ||
|
||
<TabItem value="cloud ide" label="dbt Cloud IDE"> | ||
|
||
:::info | ||
You can create metrics using MetricFlow in the dbt Cloud IDE and run the [dbt sl validate](/docs/build/validation#validations-command) command. Support for running more MetricFlow commands in the IDE will be available soon. | ||
::: | ||
In dbt Cloud, run MetricFlow commands directly in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) or in the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation). | ||
|
||
For dbt Cloud CLI users, MetricFlow commands are embedded in the dbt Cloud CLI, which means you can immediately run them once you install the dbt Cloud CLI and don't need to install MetricFlow separately. You don't need to manage versioning because your dbt Cloud account will automatically manage the versioning for you. | ||
</TabItem> | ||
|
||
<TabItem value="core" label="dbt Core"> | ||
|
||
:::tip Use dbt Cloud CLI for semantic layer development | ||
|
||
You can use the dbt Cloud CLI for the experience in defining and querying metrics in your dbt project. | ||
|
||
A benefit to using the dbt Cloud is that you won't need to manage versioning — your dbt Cloud account will automatically manage the versioning. | ||
::: | ||
<TabItem value="core" label="MetricFlow with dbt Core"> | ||
|
||
You can install [MetricFlow](https://github.com/dbt-labs/metricflow#getting-started) from [PyPI](https://pypi.org/project/dbt-metricflow/). You need to use `pip` to install MetricFlow on Windows or Linux operating systems: | ||
|
||
|
@@ -54,31 +39,34 @@ You can install [MetricFlow](https://github.com/dbt-labs/metricflow#getting-star | |
|
||
**Note**, you'll need to manage versioning between dbt Core, your adapter, and MetricFlow. | ||
|
||
</TabItem> | ||
Something to note, MetricFlow `mf` commands return an error if you have a Metafont latex package installed. To run `mf` commands, uninstall the package. | ||
|
||
</TabItem> | ||
</Tabs> | ||
|
||
Something to note, MetricFlow `mf` commands return an error if you have a Metafont latex package installed. To run `mf` commands, uninstall the package. | ||
|
||
## MetricFlow commands | ||
|
||
MetricFlow provides the following commands to retrieve metadata and query metrics. | ||
|
||
<Tabs> | ||
<TabItem value="cloud" label="Commands for dbt Cloud CLI"> | ||
<TabItem value="cloudcommands" label="Commands for dbt Cloud"> | ||
|
||
You can use the `dbt sl` prefix before the command name to execute them in the dbt Cloud CLI. For example, to list all metrics, run `dbt sl list metrics`. For a complete list of the MetricFlow commands and flags, run the `dbt sl --help` command in your terminal. | ||
You can use the `dbt sl` prefix before the command name to execute them in the dbt Cloud IDE or dbt Cloud CLI. For example, to list all metrics, run `dbt sl list metrics`. For a complete list of the MetricFlow commands and flags, run the `dbt sl --help` command in your terminal. | ||
|
||
- [`list`](#list) — Retrieves metadata values. | ||
- [`list metrics`](#list-metrics) — Lists metrics with dimensions. | ||
- [`list dimensions`](#list) — Lists unique dimensions for metrics. | ||
- [`list dimension-values`](#list-dimension-values) — List dimensions with metrics. | ||
- [`list entities`](#list-entities) — Lists all unique entities. | ||
- [`list saved-queries`](#list-saved-queries) — Lists available saved queries. Use the `--show-exports` flag to display each export listed under a saved query. | ||
- [`query`](#query) — Query metrics, saved queries, and dimensions you want to see in the command line interface. Refer to [query examples](#query-examples) to help you get started. | ||
- [`export`](#export) — Runs exports for a singular saved query for testing and generating exports in your development environment. You can also use the `--select` flag to specify particular exports from a saved query. | ||
- [`export-all`](#export-all) — Runs exports for multiple saved queries at once, saving time and effort. | ||
- [`validate`](#validate) — Validates semantic model configurations. | ||
Note: Only the `list`, `query`, and `validate` commands are available in the dbt Cloud IDE. | ||
|
||
| <div style={{width:'250px'}}>Command</div> | <div style={{width:'100px'}}>Description</div> | dbt Cloud IDE | dbt Cloud CLI | | ||
|---------|-------------|---------------|---------------| | ||
| [`list`](#list) | Retrieves metadata values. | ✅ | ✅ | | ||
| [`list metrics`](#list-metrics) | Lists metrics with dimensions. | ✅ | ✅ | | ||
| [`list saved-queries`](#list-saved-queries) | Lists available saved queries. Use the `--show-exports` flag to display each export listed under a saved query. | ✅ | ✅ | | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. you can also do |
||
| [`query`](#query) | Query metrics, saved queries, and dimensions you want to see in the command line interface. Refer to [query examples](#query-examples) to help you get started. | ✅ | ✅ | | ||
| [`validate`](#validate) | Validates semantic model configurations. | ✅ | ✅ | | ||
| [`export`](#export) | Runs exports for a singular saved query for testing and generating exports in your development environment. You can also use the `--select` flag to specify particular exports from a saved query. | ❌ | ✅ | | ||
mirnawong1 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| [`export-all`](#export-all) | Runs exports for multiple saved queries at once, saving time and effort. | ❌ | ✅ | | ||
| [`list dimensions`](#list) | Lists unique dimensions for metrics. | ❌ | ✅ | | ||
| [`list dimension-values`](#list-dimension-values) | List dimensions with metrics. | ❌ | ✅ | | ||
| [`list entities`](#list-entities) | Lists all unique entities. | ❌ | ✅ | | ||
|
||
<!--below commands aren't supported in dbt cloud yet | ||
- [`health-checks`](#health-checks) — Performs data platform health check. | ||
|
@@ -99,7 +87,7 @@ Check out the following video for a short video demo of how to query or preview | |
|
||
</TabItem> | ||
|
||
<TabItem value="core" label="Commands for dbt Core"> | ||
<TabItem value="corecommands" label="Commands for dbt Core"> | ||
|
||
Use the `mf` prefix before the command name to execute them in dbt Core. For example, to list all metrics, run `mf list metrics`. | ||
|
||
|
@@ -502,8 +490,6 @@ The following tabs present additional query examples, like exporting to a CSV. S | |
|
||
<Tabs> | ||
|
||
|
||
|
||
<TabItem value="eg6" label="--compile/--explain flag"> | ||
|
||
Add `--compile` (or `--explain` for dbt Core users) to your query to view the SQL generated by MetricFlow. | ||
|
@@ -522,24 +508,24 @@ mf query --metrics order_total --group-by metric_time,is_food_order --limit 10 - | |
```bash | ||
✔ Success 🦄 - query completed after 0.28 seconds | ||
🔎 SQL (remove --compile to see data or add --show-dataflow-plan to see the generated dataflow plan): | ||
SELECT | ||
select | ||
metric_time | ||
, is_food_order | ||
, SUM(order_cost) AS order_total | ||
FROM ( | ||
SELECT | ||
cast(ordered_at as date) AS metric_time | ||
, sum(order_cost) as order_total | ||
from ( | ||
select | ||
cast(ordered_at as date) as metric_time | ||
, is_food_order | ||
, order_cost | ||
FROM ANALYTICS.js_dbt_sl_demo.orders orders_src_1 | ||
WHERE cast(ordered_at as date) BETWEEN CAST('2017-08-22' AS TIMESTAMP) AND CAST('2017-08-27' AS TIMESTAMP) | ||
from analytics.js_dbt_sl_demo.orders orders_src_1 | ||
where cast(ordered_at as date) between cast('2017-08-22' as timestamp) and cast('2017-08-27' as timestamp) | ||
) subq_3 | ||
WHERE is_food_order = True | ||
GROUP BY | ||
where is_food_order = True | ||
group by | ||
metric_time | ||
, is_food_order | ||
ORDER BY metric_time DESC | ||
LIMIT 10 | ||
order by metric_time desc | ||
limit 10 | ||
``` | ||
|
||
</TabItem> | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -54,7 +54,7 @@ Before you're able to run exports in development or production, you'll need to m | |
|
||
There are two ways to run an export: | ||
|
||
- [Run exports in development](#exports-in-development) using the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) to test the output before production (You can configure exports in the dbt Cloud IDE, however running them directly in the IDE isn't supported yet). | ||
- [Run exports in development](#exports-in-development) using the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) to test the output before production (You can configure exports in the dbt Cloud IDE, however running them directly in the IDE isn't supported yet). Confirm | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. can we confirm that exports can only be run in the cloud cli for development? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed! |
||
- [Run exports in production](#exports-in-production) using the [dbt Cloud job scheduler](/docs/deploy/job-scheduler) to write these queries within your data platform. | ||
|
||
## Exports in development | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So the sad reality is that at the moment
dbt --help
does not work in IDE due to how they process logs. If you dodbt build --help
ordbt sl list metrics --help
or any other help command, it will just output nothing...There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might be worth talking to someone from the IDE team if we can get support for that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For now tho, the user's best bet is to refer to the docs.