diff --git a/docs/content/dagster-cloud/insights.mdx b/docs/content/dagster-cloud/insights.mdx new file mode 100644 index 0000000000000..d9cbc37f8087c --- /dev/null +++ b/docs/content/dagster-cloud/insights.mdx @@ -0,0 +1,105 @@ +--- +title: Dagster Cloud Insights +description: "Visibility into historical usage and cost metrics." + +platform_type: "cloud" +--- + +# Dagster Cloud Insights + + + This feature is considered experimental. To get access to + Insights please request access in the [#dagster-insights](https://dagster.slack.com/archives/C05V7GETFSQ) channel in Slack. + + +Insights is Dagster Cloud feature that provides visibility into historical usage and cost metrics such as run duration, credit usage and failures. This feature is available as a top level tab in the Dagster Cloud UI: + + + +The Insights page shows a list of metrics in the left panel. For each metric the daily, weekly or monthly aggregated values are shown in a graph in the main panel. As of October 2023 the metrics are update once a day. + +## External metrics + +External metrics such as Snowflake credits spent can be integrated in the Dagster Insights UI. The [`dagster-cloud`](https://pypi.org/project/dagster-cloud/) package contains utilities for capturing and submitting external metrics about data operations to Dagster Cloud via an API. + +### How to enable Snowflake and dbt with Insights + +If you use dbt to materialize tables in Snowflake, you can use these instructions to integrate Snowflake metrics into the Insights UI. + +#### Step 1 - Instrument your dbt asset definition + +You need `dagster-cloud` version 1.5.1 or newer. Instrument the dagster `@dbt_assets` function with `dbt_with_snowflake_insights`. + +This passes through all the underlying events and in addition emits an `AssetObservation` for each materialization. This observation contains the dbt invocation id and unique id that get recorded in the Dagster event log. + +```python +from dagster_cloud.dagster_insights import dbt_with_snowflake_insights +@dbt_assets(...) +def my_asset(context: AssetExecutionContext): + # Typically you have a `yield from dbt_resource.cli(...)`. + # Wrap the original call with `dbt_with_snowflake_insights` as below. + dbt_cli_invocation = dbt_resource.cli(["build"], context=context) + yield from dbt_with_snowflake_insights(context, dbt_cli_invocation) +``` + +#### Step 2 - Update your dbt_project.yml + +Add the following to your `dbt_project.yml`: + +```yaml +query-comment: + comment: "snowflake_dagster_dbt_v1_opaque_id[[[{{ node.unique_id }}:{{ invocation_id }}]]]" + append: true +``` + +This adds a comment to each query recorded in the `query_history` table in Snowflake. The comment contains the dbt unique id and invocation id. Here `append: true` is important since Snowflake strips leading comments. + +#### Step 3 - Create a metrics ingestion pipeline + +Create a Dagster pipeline that joins asset observation events with the Snowflake query history and calls the Dagster Cloud ingestion API. This needs a Snowflake resource that can query `query_history`. You can use a pre-defined pipeline as below: + +```python +from datetime import date +from dagster_snowflake import SnowflakeResource +from dagster import Definition, EnvVar + +from dagster_cloud.dagster_insights import ( + create_snowflake_insights_asset_and_schedule, +) + +snowflake_insights_definitions = create_snowflake_insights_asset_and_schedule( + date(2023, 10, 5), + allow_partial_partitions=True, + dry_run=False, + snowflake_resource_key="snowflake_insights", +) + +defs = Definitions( + assets=[..., *snowflake_insights_definitions.assets], + schedules=[..., snowflake_insights_deifnitions.schedule], + resources={ + ..., + "snowflake_insights": SnowflakeResource( + account=EnvVar("SNOWFLAKE_PURINA_ACCOUNT"), + user=EnvVar("SNOWFLAKE_PURINA_USER"), + password=EnvVar("SNOWFLAKE_PURINA_PASSWORD"), + ), + } +) +``` + +The `snowflake_resource_key` is a SnowflakeResource that has access to the `query_history` table. Once the pipeline runs, Snowflake credits should be visible in the Insights tab: + + + +--- diff --git a/docs/next/public/images/dagster-cloud/insights/insights-snowflake.png b/docs/next/public/images/dagster-cloud/insights/insights-snowflake.png new file mode 100644 index 0000000000000..4e05c019f6eea Binary files /dev/null and b/docs/next/public/images/dagster-cloud/insights/insights-snowflake.png differ diff --git a/docs/next/public/images/dagster-cloud/insights/insights-tab.png b/docs/next/public/images/dagster-cloud/insights/insights-tab.png new file mode 100644 index 0000000000000..834e367a7dda1 Binary files /dev/null and b/docs/next/public/images/dagster-cloud/insights/insights-tab.png differ