Skip to content

Commit

Permalink
[docs] [insights] - Update/format insights changes (#17125)
Browse files Browse the repository at this point in the history
## Summary & Motivation

This builds on #17119. It adds a note about an Enterprise only feature
to relevant guides and formats/edits some of the concept page copy.

## How I Tested These Changes

👀

---------

Co-authored-by: elementljarredc <[email protected]>
Co-authored-by: Shalabh Chaturvedi <[email protected]>
  • Loading branch information
3 people authored Oct 10, 2023
1 parent 677a68f commit 3ec4767
Show file tree
Hide file tree
Showing 3 changed files with 31 additions and 30 deletions.
58 changes: 29 additions & 29 deletions docs/content/dagster-cloud/insights.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,36 @@ platform_type: "cloud"
# Dagster Cloud Insights

<Note>
This feature is considered <strong>experimental</strong>. Request access using
the&nbsp;
<a href="https://dagster.slack.com/archives/C05V7GETFSQ">#dagster-insights</a>
&nbsp;channel in the Dagster Slack.
This feature is considered <strong>experimental</strong>.
</Note>

Using Dagster Cloud Insights, you can gain visibility into historical usage and cost metrics such as Dagster Cloud run duration, credit usage, and failures. You can also [integrate other external metrics](#integrating-external-metrics) into the Insights UI and [export metrics from Dagster Cloud](#exporting-metrics).

---

## About Insights

Launched in October 2023, Insights is currently an **experimental feature.** This means we’re still collecting feedback on how Insights works and could make changes based on that feedback.

**Note**: These changes may include breaking changes to some aspects of the feature, though we will attempt to avoid it.

### Getting access

Currently, Insights can only be enabled by the Dagster team. To request access, you can:

- Reach out to us directly in your company's Slack channel
- Request access in the Dagster Slack <a href="https://dagster.slack.com/archives/C05V7GETFSQ" target="blank">#dagster-insights</a> channel

**Note**: While visualizing and exploring [Dagster-generated metrics](#available-metrics) within the UI will be available to all cloud users, [integrating external metrics](#integrating-external-metrics) into Insights is an **Enterprise-only feature**.

### Future plans

As part of the experimental release, out-of-the-box utilities for capturing external metadata are limited to dbt + Snowflake pipelines that create [Software-defined Assets](/concepts/assets/software-defined-assets). Refer to the [Integrating Snowflake + dbt into Insights guide](/dagster-cloud/insights/integrating-snowflake-and-dbt) for more info.

Additional data sources and capabilities for capturing arbitrary metadata, as well as native tooling for enabling these capabilities on pipelines using ops and jobs, will be released in the coming weeks.

---

## Accessing Insights in the Dagster UI

To access this feature, click **Insights** in the top navigation bar in the UI:
Expand All @@ -35,9 +55,7 @@ The left navigation panel on this page contains a list of [available metrics](#a

These metrics are updated on a daily basis.

---

## Available metrics
### Available metrics

<table
className="table"
Expand Down Expand Up @@ -134,30 +152,12 @@ These metrics are updated on a daily basis.

---

## Integrating external metrics
## Uses

External metrics, such as Snowflake credits, can be integrated into Dagster Insights. Refer to the [Integrating external metrics into Dagster Cloud guide](/dagster-cloud/insights/integrating-external-metrics) for details.
### Integrating external metrics

---
External metrics, such as Snowflake credits, can be integrated into Dagster Insights. Refer to the [Integrating external metrics into Dagster Cloud guide](/dagster-cloud/insights/integrating-external-metrics) for details.

## Exporting metrics
### Exporting metrics

Metrics in Dagster Cloud Insights can be exported using a GraphQL API endpoint. Refer to the [Exporting Insights metrics from Dagster Cloud guide](/dagster-cloud/insights/exporting-insights-metrics) for details.

---

## Why is Insights experimental?

Dagster Insights are a new feature that we launched in Oct 2023. The experimental label means we’re still working with design partners to collect feedback on how it works, and could make changes based on that feedback. It’s possible that we could make breaking changes to some of aspects of the feature, though we’re going to do our best to avoid that.

## How to get access

The Dagster Insights is an experimental feature, which means that it can only be enabled on request from our team. To get access to it, please reach out to us directly through your company slack channel or by joining #dagster-insights to request it.

While visualizing and exploring Dagster generated metrics within the UI will be available to all cloud users, capturing external metadata in our metrics store is an **Enterprise-only feature**

## Future plans

As part of the experimental release, out of the box utilities for capturing external metadata are limited to dbt+Snowflake pipelines creating software-defined assets.

We plan to quickly follow up with additional data sources and capabilities for capturing arbitrary metadata, as well as native tooling for enabling these capabilities on pipelines using the ops / jobs framework in the coming weeks.
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ External metrics, such as Snowflake credits, can be integrated into the Dagster

Before you start, note that:

- This is a Dagster Cloud Enterprise feature
- Up to two million individual data points may be added to Insights, per month
- External metrics data will only be retained for 90 days

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ If you use dbt to materialize tables in Snowflake, use this guide to integrate S

To complete the steps in this guide, you'll need:

- A Dagster Cloud account
- A Dagster Cloud account on the Enterprise plan
- Access to the [Dagster Cloud Insights feature](/dagster-cloud/insights)
- To install the following libraries:

Expand Down

1 comment on commit 3ec4767

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs ready!

✅ Preview
https://dagster-docs-6ssc98yzd-elementl.vercel.app
https://master.dagster.dagster-docs.io

Built with commit 3ec4767.
This pull request is being automatically deployed with vercel-action

Please sign in to comment.