Docs updates

docs/best-practices/detection-and-coverage.mdx

@@ -0,0 +1,150 @@
+---
+title: "Detection and coverage"
+---
+
+In Elementary you can detect data issues by combining data validations (as dbt tests, custom SQL) and anomaly detection monitors.
+
+As you expand your coverage, it's crucial to balance between coverage and meaningful detections. While it may seem attractive to implement extensive monitoring throughout your data infrastructure, this approach is often suboptimal. Excessive failures can lead to alerts fatigue, potentially causing them to overlook significant issues. Additionally, such approach will incur unnecessary compute costs.
+
+In this section we will cover the available tests in Elementary, recommended tests for common use cases, and how to use the data quality dimensions framework to improve coverage.
+
+## Supported data tests and monitors
+
+Elementary detection includes:
+
+- Data tests - Validate an explicit expectation, and fail if it is not met.
+  - Example: validate there are no null values in a column.
+- Anomaly detection monitors - Track a data quality metric over time, and fail if there is an anomaly comparing to previous values and trend.
+  - Example: track the rate of null values in a column over time, fail if there is a spike.
+
+### Data tests -
+
+- dbt tests - Built in dbt tests (`not_null`, `unique`, `accepted_values`, `relationship` )
+- dbt packages - Any dbt package test, we recommend installing `dbt-utils` and `dbt-expectations` .
+- Custom SQL tests - Custom query, will pass if no results and fail if any results are returned.
+
+### Anomaly detection monitors -
+
+Elementary offers two types of anomaly detection monitors:
+
+- **Automated Monitors** - Out-of-the-box volume and freshness monitors activated automatically, that query metadata only.
+- **Opt-in anomaly detection tests** - Monitors that query raw data and require configuration.
+
+<Check>
+### Recommendations
+
+- Deploy the packages dbt-utils and dbt-expectations in your dbt projects, to enrich your available tests
+- Refer to the [dbt test hub](https://www.elementary-data.com/dbt-test-hub) by Elementary, to explore available tests by use case
+</Check>
+
+## Fine-tuning automated monitors
+
+As soon as you connect Elementary Cloud Platform to your data warehouse, a backfill process will begin to collect historical metadata. Within an average of a few hours, your automated monitors will be operational. By default, Elementary collects at least 21 days of historical metadata.
+
+You can fine tune the [**configuration**](https://docs.elementary-data.com/features/anomaly-detection/monitors-configuration) and [**provide feedback**](https://docs.elementary-data.com/features/anomaly-detection/monitors-feedback) to adjust the detection to your needs.
+
+You can read here about how to interpret the result, and what are the available setting of each monitor:
+
+- [Automated Freshness](https://docs.elementary-data.com/features/anomaly-detection/automated-freshness)
+- [Automated Volume](https://docs.elementary-data.com/features/anomaly-detection/automated-volume)
+
+## Common testing use cases
+
+We have the following recommendations for testing different data assets:
+
+### Data sources
+
+To detect issues in sources updates, you should monitor volume, freshness and schema:
+
+- Volume and freshness
+  - Data updates - Elementary cloud provides automated monitors for freshness and volume. **These are metadata monitors.**
+    - Updates freshness vs. data freshness - The automated freshness will detect delays in **updates**. \*\*\*\*However, sometimes the update will be on time, but the data itself will be outdated.
+  - Data freshness (advanced) - Sometimes a table can update on time, but the data itself will be outdated. If you want to validate the freshness of the raw data by relaying on the actual timestamp, you can use:
+    - Explicit treshold [freshness dbt tests](https://www.elementary-data.com/dbt-test-hub) such as `dbt_utils.recency` , or [dbt source freshness](https://docs.getdbt.com/docs/deploy/source-freshness).
+    - Elementary `event_freshness_anomalies` to detect anomalies.
+  - Data volume (advanced) - Although a table can be updated as expected, the data itself might still be imbalanced in terms of volume per specific segment. There are several tests available to monitor that:
+    - Explicit [volume expectations](https://www.elementary-data.com/dbt-test-hub) such as `expect_table_row_count_to_be_between`.
+    - Elementary `dimension_anomalies` , that will count rows grouped by a column or combination of columns and can detect drops or spikes in volume in specific subsets of the data.
+- Schema changes
+
+  - Automated schema monitors are coming soon:
+    - These monitors will detect breaking changes to the schema only for columns being consumed based on lineage.
+  - For now, we recommend defining schema tests on the sources consumed by downstream staging models.
+
+  Some validations on the data itself should be added in the source tables, to test early in the pipeline and detect when data is arriving with an issue from the source.
+
+  - Low cardinality columns / strict set of values - If there are fields with a specific set of values you expect use `accepted_values`. If you also expect a consistency in ratio of these values, use `dimension_anomalies` and group by this column.
+  - Business requirements - If you are aware of expectations specific to your business, try to enforce early to detect when issues are at the source. Some examples: `expect_column_values_to_be_between`, `expect_column_values_to_be_increasing`, `expect-column-values-to-have-consistent-casing`
+
+<Check>
+### Recommendations
+
+- Add data freshness and volume validations for relevant source tables, on top of the automated monitors (advanced)
+- Add schema tests for source tables
+</Check>
+
+### Primary / foreign key columns in your transformation models
+
+Tables should be covered with:
+
+- Unique checks on primary / foreign key columns to detect unnecessary duplications during data transformations.
+- Not null checks on primary / foreign key columns to detect missing values during data transformations.
+
+For incremental tables, it’s recommended to use a `where` clause in the tests, and only validate recent data. This will prevent running the tests on large data sets which is costly and slow.
+
+<Check>
+#### Recommendations
+
+- Add `unique` and `not_null` tests to key columns
+</Check>
+
+### Public tables
+
+As these are your data products, coverage here is highly important.
+
+- Consistency with sources (based on aggregation/primary keys)
+- Volume and freshness
+- Unique and not null checks on primary keys
+- Schema to ensure the "API" to data consumers is not broken
+- Business Metrics / KPIs
+  - Sum / max anomalies group by your critical dimensions / segments (For example - country, platform…)
+
+### Data quality dimensions framework
+
+To ensure your detection and coverage have a solid baseline, we recommend leveraging the quality dimensions framework for your critical and public assets.
+
+The quality dimensions framework divides data validation into six common dimensions:
+
+- **Completeness**: No missing values, empty values, nulls, etc.
+- **Uniqueness**: The data is unique, with no duplicates.
+- **Freshness**: The data is up to date and within the expected SLAs.
+- **Validity**: The data is in the correct format and structure.
+- **Accuracy**: The data adheres to our business requirements and constraints.
+- **Consistency**: The data is consistent from sources to targets, and from sources to where it is consumed.
+
+Elementary has already categorized all the existing tests in the dbt ecosystem, including all elementary anomaly detection monitors, into these quality dimensions and provides health scores per dimension automatically. It also shows if there are coverage gaps per dimension.
+
+We highly recommend going to the relevant quality dimension, then filtering by a business domain tag to see your coverage gaps in that domain.
+
+Example -
+
+![Data health dashboard](/pics/data-health-dashboard.webp)
+
+In this example, you can see that accuracy tests are missing for our sales domain. This means we don't know if the data in our public-facing "sales" tables adheres to our business constraints. For example, if we have an e-commerce shop where no product has a price below $100 or above $1000, we can easily add a test to validate this. Implementing validations for the main constraints in this domain will allow us to get a quality score for the accuracy level of our data.
+
+NOTE: The `Test Coverage` page in Elementary allows adding any dbt test from the ecosystem, Elementary anomaly detection monitors, and custom SQL tests. We are working on making it easier to add tests by creating a test catalog organized by quality dimensions and common use cases.
+
+Example for tests in each quality dimension -
+
+- **Completeness**:
+  - not_null, null count, null percent, missing values, empty values, column anomalies on null count, null percent, etc
+- **Uniqueness**:
+  - unique, expect_column_values_to_be_unique, expect_column_unique_value_count_to_be_between, expect_compound_columns_to_be_unique
+- **Freshness**: The data is up to date and within the expected SLAs.
+  - Elementary automated freshness monitor, dbt source freshness, dbt_utils.recency, expect_grouped_row_values_to_have_recent_data
+- **Validity**: The data is in the correct format and structure.
+  - expect_column_values_to_match_regex, expect_column_min_to_be_between, expect_column_max_to_be_between, expect_column_value_lengths_to_be_between, column anomalies on min, max, string lengths
+- **Accuracy**: The data adheres to our business requirements and constraints.
+  - expression_is_true, custom SQL
+- **Consistency**: The data is consistent from sources to targets, and from sources to where it is consumed.
+  - relationship, expect_table_row_count_to_equal_other_table, expect_table_aggregation_to_equal_other_table

docs/best-practices/governance-for-observability.mdx

@@ -0,0 +1,134 @@
+---
+title: "Governance for observability"
+---
+
+For an effective data observability process, it’s recommended to establish clear ownership, priorities and segmentation of data assets. This structure enhances governance, speeds up issue resolution, and improves data health tracking.
+
+Segmenting assets organizes data into manageable units, making monitoring and triage easier. Ownership ensures accountability, with specific individuals responsible for quality and response to incidents.
+
+## Introduction to tags, owners and subscribers
+
+### Tags
+
+As your data platform evolves and more people are maintaining it, structure and context become significantly more important. Tags are a great tool to create that context, and segment your data assets by business domains, data products, priority, etc.
+
+In Elementary tags are automatically included in alerts, and you can create rules to distribute alerts to different channels by tag. Additionally, different views in the platform can be filtered by tag, and provide a view for a subset of your data assets.
+
+- Tags for tables can be added in code at the model or folder level, and the `tags` key.
+- It’s recommended to leverage dbt directories hierarchy to set tags to entire directories (in the dbt_project.yml). Tags are aggregated, so if a specific model under the directory has a different tag, the model will have both tags.
+
+```yaml
+models:
+	analytics:
+    marketing:
+	    +tags: marketing
+	    public:
+		    +tags: marketing-public
+```
+
+- Tags for tests can be added in code or in the Elementary UI when adding a test.
+
+### Owners and subscribers
+
+The best method to reduce time to response when there is a data issue is having a clear owner that is in charge of initial triage and accountable for the asset health. In Elementary owners are automatically tagged in alerts. Additionally, different views in the platform can be filtered by owner.
+
+A data asset or test should have only one owner, but other people might want to be notified on issues. These people can be listed as subscribers, and will be automatically tagged in alerts.
+
+- If you use a valid Slack / MS teams user as owner / subscriber, they will be tagged in alerts.
+- The owner of an asset should be the person / team that is expected to respond to an issue in that asset.
+- If there are specific tests or monitors that are relevant to other people, they can be the owners of these tests.
+  For example: A data engineer is the owner of a model and will be notified on freshness, volume, and data validations issues. A data analyst added some custom SQL tests to validate business logic on this model, and he owns these tests.
+- It’s recommended to leverage dbt directories hierarchy to set owners to entire directories (in the dbt_project.yml). Owners are unique, so an owner that is defined on a model overrides the directory configuration. (Subscribers are aggregated).
+
+```yaml
+models:
+  - name: return_on_ad_spend
+    tags:
+	    - marketing-public
+	    - marketing
+    meta:
+	    owner: :"@analytics.engineer"
+      subscribers:
+      - "@marketing.data.analyst"
+      - "@another.marketing.data.analyst"
+```
+
+## Business domains & Data products
+
+- We recommend configuring the following tags for models:
+  - **Business domains** - These tags should be useful to understand what is the business context of the asset, and for stakeholders to filter and view the status of assets relevant to their business unit. Relevant examples are tags such as: `product-analytics` , `marketing` , `finance` , etc.
+  - **Data products** - Public tables that are exposed as “data products” to data consumers. These are the most important tables within a specific domain, similar to an API for an application. Public tables are usually the interface and focal point between analytics engineers and data analysts. It's crucial for both to be aware of any data issues in these tables. Relevant examples are tags such as: `product-analytics-public` , `marketing-public` , `data-science-public` , etc.
+  - Another possible implementation is using 3 types of tags -
+    - `marketing-internal` for all internal transformations on marketing data.
+    - `marketing-public` for all public-facing marketing data.
+    - `marketing` for all marketing-related data assets.
+- **Owners and subscribers -**
+
+  - Make sure to have clear ownership defined for all your public-facing tables. We also recommend adding subscribers to the relevant public tables.
+  - Usually, the owners of these public tables are the analytics engineering team, and the subscribers are the relevant data analysts who rely on the data from these tables.
+
+<Check>
+### Recommendations
+
+- Add business domain tags to public tables
+- Define owners for public facing tables
+- Add data consumers as subscribers to relevant public facing tables
+</Check>
+
+
+
+## Priorities (optional)
+
+Another useful tagging convention can be to set a tag that filters a subset of assets by their priority, so you could establish a process of response to issues with higher criticality.
+
+Decide how many levels of priority you wish to maintain, and implement by adding a `critical` tag to your critical assets, or create a `P0`, `P1` , `P2` tags for several priority levels.
+
+This will enable you to filter the results in Elementary by priority, and establish workflows such as sending `critical` alerts to Pagerduty, and the rest to Slack.
+
+<Check>
+### Recommendations
+
+- Add priorities / critical tags to tables / tests (Optional)
+- Add owners to all top priority tables / tests (Optional)
+</Check>
+
+## Data sources
+
+Many data issues are a result of a problem in the source data, so effectively monitoring source tables is significant to your pipeline health.
+
+Use tags to segment your source tables:
+
+- If multiple source tables are loaded from the same source, we recommend grouping them by tags, such as: `mongo-db-replica`, `salesforce`, `prod-postgres`, etc.
+- To make triage easier, you can also add tags of the ingestion system, such as: `fivetran`, `airflow` , `airbyte` , `kafka` , etc.
+
+Ownership and subscribers:
+
+- Usually, sources are managed by data engineers and analytics engineers are their consumers. One common way to manage this is to set data engineers as the owners and analytics engineering team members as the subscribers.
+
+```yaml
+sources:
+  - name: fivetran_salesforce_sync
+    tags:
+     - fivetran
+     - salesforce
+    meta:
+	    owner: :"@data.engineer"
+      subscribers: "@analytics.engineer"
+```
+
+<Check>
+### Recommendations
+
+- Add tags to source tables that describe the source system and / or ingestion method
+- Add owners and subscribers to source tables
+</Check>
+
+## Recommendations
+
+- Add business domain tags to public tables
+- Define owners for public facing tables
+- Add data consumers as subscribers to relevant public facing tables
+- (Optional) Add priorities / critical tags to tables / tests
+- (Optional) Add owners to all top priority tables / tests
+- Add tags to source tables that describe the source system and / or ingestion method
+- Add owners and subscribers to source tables

docs/best-practices/introduction.mdx

@@ -0,0 +1,33 @@
+---
+title: "Elementary Best Practices"
+sidebarTitle: "Introduction"
+---
+
+<Info>
+  The goal of this collection of guides is to help you effectively implement and use Elementary.
+  We'll cover best practices and provide practical tips to enhance your governance, detection,
+  coverage, response and collaboration. Whether you're new to Elementary or looking to optimize your
+  current usage, these guides will help you leverage its full potential to improve your data
+  reliability.
+</Info>
+
+<CardGroup cols={3}>
+  <Card
+    title="Governance for observability"
+    href="/best-practices/governance-for-observability"
+    icon="tower-observation"
+    iconType="solid"
+  ></Card>
+  <Card
+    title="Detection and coverage"
+    href="/best-practices/detection-and-coverage"
+    icon="sensor-triangle-exclamation"
+    iconType="solid"
+  ></Card>
+  <Card
+    title="Triage & response"
+    href="/best-practices/triage-and-response"
+    icon="circle-exclamation"
+    iconType="solid"
+  ></Card>
+</CardGroup>