Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update sl ide support #6072

Merged
merged 18 commits into from
Sep 26, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 36 additions & 50 deletions website/docs/docs/build/metricflow-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ tags: [Metrics, Semantic Layer]

Once you define metrics in your dbt project, you can query metrics, dimensions, and dimension values, and validate your configs using the MetricFlow commands.

MetricFlow allows you to define and query metrics in your dbt project in the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation), [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud), or [dbt Core](/docs/core/installation-overview). To experience the power of the universal [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) and dynamically query those metrics in downstream tools, you'll need a dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) account.
MetricFlow allows you to define and query metrics in your dbt project in the [dbt Cloud](/docs/cloud/about-develop-dbt) or [dbt Core](/docs/core/installation-overview). To experience the power of the universal [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) and dynamically query those metrics in downstream tools, you'll need a dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) account.

MetricFlow is compatible with Python versions 3.8, 3.9, 3.10, and 3.11.

Expand All @@ -18,33 +18,18 @@ MetricFlow is a dbt package that allows you to define and query metrics in your

Using MetricFlow with dbt Cloud means you won't need to manage versioning — your dbt Cloud account will automatically manage the versioning.

**dbt Cloud jobs** — MetricFlow commands aren't supported in dbt Cloud jobs yet. However, you can add MetricFlow validations with your git provider (such as GitHub Actions) by installing MetricFlow (`python -m pip install metricflow`). This allows you to run MetricFlow commands as part of your continuous integration checks on PRs.
dbt Cloud jobs support the `dbt sl validate` command to [automatically test your semantic nodes](/docs/deploy/ci-jobs#semantic-validations-in-ci). You can also add MetricFlow validations with your git provider (such as GitHub Actions) by installing MetricFlow (`python -m pip install metricflow`). This allows you to run MetricFlow commands as part of your continuous integration checks on PRs.

<Tabs>

<TabItem value="cloudcli" label="dbt Cloud CLI">
<TabItem value="cloud" label="MetricFlow with dbt Cloud">

- MetricFlow [commands](#metricflow-commands) are embedded in the dbt Cloud CLI. This means you can immediately run them once you install the dbt Cloud CLI and don't need to install MetricFlow separately.
- You don't need to manage versioning &mdash; your dbt Cloud account will automatically manage the versioning for you.

</TabItem>

<TabItem value="cloud ide" label="dbt Cloud IDE">

:::info
You can create metrics using MetricFlow in the dbt Cloud IDE and run the [dbt sl validate](/docs/build/validation#validations-command) command. Support for running more MetricFlow commands in the IDE will be available soon.
:::
In dbt Cloud, run MetricFlow commands directly in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) or in the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation).

For dbt Cloud CLI users, MetricFlow commands are embedded in the dbt Cloud CLI, which means you can immediately run them once you install the dbt Cloud CLI and don't need to install MetricFlow separately. You don't need to manage versioning because your dbt Cloud account will automatically manage the versioning for you.
</TabItem>

<TabItem value="core" label="dbt Core">

:::tip Use dbt Cloud CLI for semantic layer development

You can use the dbt Cloud CLI for the experience in defining and querying metrics in your dbt project.

A benefit to using the dbt Cloud is that you won't need to manage versioning &mdash; your dbt Cloud account will automatically manage the versioning.
:::
<TabItem value="core" label="MetricFlow with dbt Core">

You can install [MetricFlow](https://github.com/dbt-labs/metricflow#getting-started) from [PyPI](https://pypi.org/project/dbt-metricflow/). You need to use `pip` to install MetricFlow on Windows or Linux operating systems:

Expand All @@ -54,31 +39,34 @@ You can install [MetricFlow](https://github.com/dbt-labs/metricflow#getting-star

**Note**, you'll need to manage versioning between dbt Core, your adapter, and MetricFlow.

</TabItem>
Something to note, MetricFlow `mf` commands return an error if you have a Metafont latex package installed. To run `mf` commands, uninstall the package.

</TabItem>
</Tabs>

Something to note, MetricFlow `mf` commands return an error if you have a Metafont latex package installed. To run `mf` commands, uninstall the package.

## MetricFlow commands

MetricFlow provides the following commands to retrieve metadata and query metrics.

<Tabs>
<TabItem value="cloud" label="Commands for dbt Cloud CLI">
<TabItem value="cloudcommands" label="Commands for dbt Cloud">

You can use the `dbt sl` prefix before the command name to execute them in the dbt Cloud CLI. For example, to list all metrics, run `dbt sl list metrics`. For a complete list of the MetricFlow commands and flags, run the `dbt sl --help` command in your terminal.
You can use the `dbt sl` prefix before the command name to execute them in the dbt Cloud IDE or dbt Cloud CLI. For example, to list all metrics, run `dbt sl list metrics`. For a complete list of the MetricFlow commands and flags, run the `dbt sl --help` command in your terminal.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the sad reality is that at the moment dbt --help does not work in IDE due to how they process logs. If you do dbt build --help or dbt sl list metrics --help or any other help command, it will just output nothing...

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be worth talking to someone from the IDE team if we can get support for that.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For now tho, the user's best bet is to refer to the docs.


- [`list`](#list) &mdash; Retrieves metadata values.
- [`list metrics`](#list-metrics) &mdash; Lists metrics with dimensions.
- [`list dimensions`](#list) &mdash; Lists unique dimensions for metrics.
- [`list dimension-values`](#list-dimension-values) &mdash; List dimensions with metrics.
- [`list entities`](#list-entities) &mdash; Lists all unique entities.
- [`list saved-queries`](#list-saved-queries) &mdash; Lists available saved queries. Use the `--show-exports` flag to display each export listed under a saved query.
- [`query`](#query) &mdash; Query metrics, saved queries, and dimensions you want to see in the command line interface. Refer to [query examples](#query-examples) to help you get started.
- [`export`](#export) &mdash; Runs exports for a singular saved query for testing and generating exports in your development environment. You can also use the `--select` flag to specify particular exports from a saved query.
- [`export-all`](#export-all) &mdash; Runs exports for multiple saved queries at once, saving time and effort.
- [`validate`](#validate) &mdash; Validates semantic model configurations.
Note: Only the `list`, `query`, and `validate` commands are available in the dbt Cloud IDE.

| <div style={{width:'250px'}}>Command</div> | <div style={{width:'100px'}}>Description</div> | dbt Cloud IDE | dbt Cloud CLI |
|---------|-------------|---------------|---------------|
| [`list`](#list) | Retrieves metadata values. | ✅ | ✅ |
| [`list metrics`](#list-metrics) | Lists metrics with dimensions. | ✅ | ✅ |
| [`list saved-queries`](#list-saved-queries) | Lists available saved queries. Use the `--show-exports` flag to display each export listed under a saved query. | ✅ | ✅ |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can also do --show-parameters to show the full query parameters each saved query uses.

| [`query`](#query) | Query metrics, saved queries, and dimensions you want to see in the command line interface. Refer to [query examples](#query-examples) to help you get started. | ✅ | ✅ |
| [`validate`](#validate) | Validates semantic model configurations. | ✅ | ✅ |
| [`export`](#export) | Runs exports for a singular saved query for testing and generating exports in your development environment. You can also use the `--select` flag to specify particular exports from a saved query. | ❌ | ✅ |
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
| [`export-all`](#export-all) | Runs exports for multiple saved queries at once, saving time and effort. | ❌ | ✅ |
| [`list dimensions`](#list) | Lists unique dimensions for metrics. | ❌ | ✅ |
| [`list dimension-values`](#list-dimension-values) | List dimensions with metrics. | ❌ | ✅ |
| [`list entities`](#list-entities) | Lists all unique entities. | ❌ | ✅ |

<!--below commands aren't supported in dbt cloud yet
- [`health-checks`](#health-checks) &mdash; Performs data platform health check.
Expand All @@ -99,7 +87,7 @@ Check out the following video for a short video demo of how to query or preview

</TabItem>

<TabItem value="core" label="Commands for dbt Core">
<TabItem value="corecommands" label="Commands for dbt Core">

Use the `mf` prefix before the command name to execute them in dbt Core. For example, to list all metrics, run `mf list metrics`.

Expand Down Expand Up @@ -502,8 +490,6 @@ The following tabs present additional query examples, like exporting to a CSV. S

<Tabs>



<TabItem value="eg6" label="--compile/--explain flag">

Add `--compile` (or `--explain` for dbt Core users) to your query to view the SQL generated by MetricFlow.
Expand All @@ -522,24 +508,24 @@ mf query --metrics order_total --group-by metric_time,is_food_order --limit 10 -
```bash
✔ Success 🦄 - query completed after 0.28 seconds
🔎 SQL (remove --compile to see data or add --show-dataflow-plan to see the generated dataflow plan):
SELECT
select
metric_time
, is_food_order
, SUM(order_cost) AS order_total
FROM (
SELECT
cast(ordered_at as date) AS metric_time
, sum(order_cost) as order_total
from (
select
cast(ordered_at as date) as metric_time
, is_food_order
, order_cost
FROM ANALYTICS.js_dbt_sl_demo.orders orders_src_1
WHERE cast(ordered_at as date) BETWEEN CAST('2017-08-22' AS TIMESTAMP) AND CAST('2017-08-27' AS TIMESTAMP)
from analytics.js_dbt_sl_demo.orders orders_src_1
where cast(ordered_at as date) between cast('2017-08-22' as timestamp) and cast('2017-08-27' as timestamp)
) subq_3
WHERE is_food_order = True
GROUP BY
where is_food_order = True
group by
metric_time
, is_food_order
ORDER BY metric_time DESC
LIMIT 10
order by metric_time desc
limit 10
```

</TabItem>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ Nice job, you're ready to start developing and building models 🎉!
- If a model or test fails, dbt Cloud makes it easy for you to view and download the run logs for your dbt invocations to fix the issue.
- Use dbt's [rich model selection syntax](/reference/node-selection/syntax) to [run dbt commands](/reference/dbt-commands) directly within dbt Cloud.
- Starting from dbt v1.6, leverage [environments variables](/docs/build/environment-variables#special-environment-variables) to dynamically use the Git branch name. For example, using the branch name as a prefix for a development schema.
- Run [MetricFlow commands](/docs/build/metricflow-commands) to create and manage metrics in your project with the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl).

- **Generate your YAML configurations with dbt Assist** <Lifecycle status="beta"/> &mdash; [dbt Assist](/docs/cloud/dbt-assist) is a powerful artificial intelligence (AI) co-pilot feature that helps automate development in dbt Cloud. It generates documentation and tests for your dbt SQL models directly in the dbt Cloud IDE, with a click of a button, and helps you accomplish more in less time. Available for dbt Cloud Enterprise plans.

Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/use-dbt-semantic-layer/exports.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ Before you're able to run exports in development or production, you'll need to m

There are two ways to run an export:

- [Run exports in development](#exports-in-development) using the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) to test the output before production (You can configure exports in the dbt Cloud IDE, however running them directly in the IDE isn't supported yet).
- [Run exports in development](#exports-in-development) using the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) to test the output before production (You can configure exports in the dbt Cloud IDE, however running them directly in the IDE isn't supported yet). Confirm
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we confirm that exports can only be run in the cloud cli for development?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed!

- [Run exports in production](#exports-in-production) using the [dbt Cloud job scheduler](/docs/deploy/job-scheduler) to write these queries within your data platform.

## Exports in development
Expand Down
4 changes: 1 addition & 3 deletions website/docs/guides/sl-snowflake-qs.md
Original file line number Diff line number Diff line change
Expand Up @@ -339,9 +339,7 @@ If you used Partner Connect, you can skip to [initializing your dbt project](#in
<Snippet path="tutorial-managed-repo" />

## Initialize your dbt project and start developing
This guide assumes you use the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) to develop your dbt project and define metrics. However, the dbt Cloud IDE doesn't support using [MetricFlow commands](/docs/build/metricflow-commands) to query or preview metrics (support coming soon).

To query and preview metrics in your development tool, you can use the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) to run the [MetricFlow commands](/docs/build/metricflow-commands).
This guide assumes you use the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) to develop your dbt project, define metrics, and query and preview metrics using [MetricFlow commands](/docs/build/metricflow-commands).

Now that you have a repository configured, you can initialize your project and start development in dbt Cloud using the IDE:

Expand Down
6 changes: 4 additions & 2 deletions website/snippets/_sl-test-and-query-metrics.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
To work with metrics in dbt, you have several tools to validate or run commands. Here's how you can test and query metrics depending on your setup:

- [**dbt Cloud IDE users**](#dbt-cloud-ide-users) &mdash; Currently, running MetricFlow commands directly in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) isn't supported, but is coming soon. You can view metrics visually through the DAG in the **Lineage** tab without directly running commands.
- [**dbt Cloud IDE users**](#dbt-cloud-ide-users) &mdash; Run [MetricFlow commands](/docs/build/metricflow-commands#metricflow-commands) directly in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) to query/preview metrics. View metrics visually in the **Lineage** tab.
- [**dbt Cloud CLI users**](#dbt-cloud-cli-users) &mdash; The [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) enables you to run [MetricFlow commands](/docs/build/metricflow-commands#metricflow-commands) to query and preview metrics directly in your command line interface.
- **dbt Core users** &mdash; Use the MetricFlow CLI for command execution. While this guide focuses on dbt Cloud users, dbt Core users can find detailed MetricFlow CLI setup instructions in the [MetricFlow commands](/docs/build/metricflow-commands#metricflow-commands) page. Note that to use the dbt Semantic Layer, you need to have a [Team or Enterprise account](https://www.getdbt.com/).

Alternatively, you can run commands with SQL client tools like DataGrip, DBeaver, or RazorSQL.

### dbt Cloud IDE users

You can view your metrics in the dbt Cloud IDE by viewing them in the **Lineage** tab. The dbt Cloud IDE **Status button** (located in the bottom right of the editor) displays an **Error** status if there's an error in your metric or semantic model definition. You can click the button to see the specific issue and resolve it.
You can use the `dbt sl` prefix before the command name to execute them in dbt Cloud. For example, to list all metrics, run `dbt sl list metrics`. For a complete list of the MetricFlow commands available in the dbt Cloud IDE, refer to the [MetricFlow commands](/docs/build/metricflow-commands#metricflow-commandss) page.

The dbt Cloud IDE **Status button** (located in the bottom right of the editor) displays an **Error** status if there's an error in your metric or semantic model definition. You can click the button to see the specific issue and resolve it.

Once viewed, make sure you commit and merge your changes in your project.

Expand Down
Loading