Data quality automatic dashboard #4785

sushanthkumar2004 · 2024-07-15T19:08:33Z

Issue #, if available:

Description of changes: Add a new automatic dashboard generation parameter to the DefaultModelMonitor and ModelQualityMonitor. Automatically generate dashboards for data quality model monitoring jobs and model quality monitoring jobs.

Testing done: Unit tests verify that the dashboard is generated correctly and all error-handling in dashboard parameters are handled correctly via a Mock Boto3 client. The integration tests verify that dashboard publishing on Cloudwatch is successful with the real Boto3 client.

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

zicanl-amazon · 2024-07-15T19:43:33Z

src/sagemaker/model_monitor/dashboards.py

+import json
+
+
+class Variable:


Is it possible to use a better name instead of Variable?

I'll call it DashboardVariable, DashboardWidget, DashboardWidgetProperty to keep consistency.

zicanl-amazon · 2024-07-15T19:53:54Z

src/sagemaker/model_monitor/model_monitoring.py

+                logger.error(message)
+                raise ValueError(message)
+
+            try:


Why do we need a try except expression here?

We need it since if the dashboard exists we don't want to overwrite it.

sage-maker · 2024-08-07T19:03:24Z

Approved workflow

tests/unit/sagemaker/monitor/test_model_monitoring.py

tests/integ/test_model_quality_monitor.py

tests/unit/sagemaker/monitor/test_model_monitoring.py

…eate_monitoring_schedule with error handling, TODO: modify update_monitoring_schedule accordingly and add unit tests

…nit tests

…_data_quality_dashboard()

… code

…nstead of endpoint name, the user provides an EndpointInput object

…to update_monitoring_schedule

…to update_monitoring_schedule methods for Data Quality and Model Quality

…Error

…oards are published using boto3

… so that only features from the specific monitoring schedule and endpoint are loaded

… modify test_dashboard_methods to take into account the new filters in the dashboard variables

sushanthkumar2004 requested a review from a team as a code owner July 15, 2024 19:08

sushanthkumar2004 requested a review from ptkab July 15, 2024 19:08

sushanthkumar2004 had a problem deploying to manual-approval July 15, 2024 19:08 — with GitHub Actions Error

zicanl-amazon reviewed Jul 15, 2024

View reviewed changes

sushanthkumar2004 had a problem deploying to manual-approval July 19, 2024 03:33 — with GitHub Actions Error

sushanthkumar2004 had a problem deploying to manual-approval July 19, 2024 03:48 — with GitHub Actions Error

sushanthkumar2004 had a problem deploying to manual-approval July 22, 2024 20:50 — with GitHub Actions Error

sushanthkumar2004 had a problem deploying to manual-approval July 22, 2024 20:52 — with GitHub Actions Error

sushanthkumar2004 had a problem deploying to manual-approval July 23, 2024 01:10 — with GitHub Actions Error

sushanthkumar2004 had a problem deploying to manual-approval July 23, 2024 03:27 — with GitHub Actions Error

sushanthkumar2004 had a problem deploying to manual-approval July 24, 2024 06:34 — with GitHub Actions Error

sushanthkumar2004 had a problem deploying to manual-approval July 25, 2024 02:02 — with GitHub Actions Error

sushanthkumar2004 had a problem deploying to manual-approval July 27, 2024 00:34 — with GitHub Actions Error

sushanthkumar2004 had a problem deploying to manual-approval July 30, 2024 20:53 — with GitHub Actions Error

sushanthkumar2004 had a problem deploying to manual-approval July 30, 2024 20:54 — with GitHub Actions Error

sushanthkumar2004 had a problem deploying to manual-approval July 31, 2024 03:37 — with GitHub Actions Error

sushanthkumar2004 had a problem deploying to manual-approval August 1, 2024 20:04 — with GitHub Actions Error

nileshvd previously approved these changes Aug 1, 2024

View reviewed changes

sushanthkumar2004 dismissed nileshvd’s stale review via f985a05 August 2, 2024 06:52

sushanthkumar2004 had a problem deploying to manual-approval August 2, 2024 06:52 — with GitHub Actions Error

zicanl-amazon approved these changes Aug 7, 2024

View reviewed changes

sushanthkumar2004 temporarily deployed to manual-approval August 7, 2024 21:33 — with GitHub Actions Inactive

sage-maker previously approved these changes Aug 7, 2024

View reviewed changes

sage-maker temporarily deployed to manual-approval August 7, 2024 22:03 — with GitHub Actions Inactive

sage-maker had a problem deploying to manual-approval August 7, 2024 22:59 — with GitHub Actions Error

sushanthkumar2004 dismissed sage-maker’s stale review via 3673a40 August 7, 2024 23:16

sushanthkumar2004 had a problem deploying to manual-approval August 7, 2024 23:16 — with GitHub Actions Error

sage-maker temporarily deployed to manual-approval August 8, 2024 00:00 — with GitHub Actions Inactive

sushanthkumar2004 had a problem deploying to manual-approval August 8, 2024 00:33 — with GitHub Actions Error

sushanthkumar2004 had a problem deploying to manual-approval August 8, 2024 00:38 — with GitHub Actions Error

sage-maker reviewed Aug 8, 2024

View reviewed changes

sushanthkumar2004 had a problem deploying to manual-approval August 8, 2024 00:46 — with GitHub Actions Error

sage-maker temporarily deployed to manual-approval August 8, 2024 01:05 — with GitHub Actions Inactive

Sushanth Sathish Kumar added 20 commits August 22, 2024 23:31

feature: Added automatic dashboard feature for DefaultModelMonitor cr…

14dad53

…eate_monitoring_schedule with error handling, TODO: modify update_monitoring_schedule accordingly and add unit tests

tests: Verify correctness of dashboard class and methods with basic u…

ee2a606

…nit tests

fix: fix dashboard tests, TODO: fix assertion error in test_automatic…

cfeb219

…_data_quality_dashboard()

feat: Add ability to create a model monitoring dashboard and refactor…

83661cf

… code

chore: Add docstrings to functions and add check for the case where i…

edc830c

…nstead of endpoint name, the user provides an EndpointInput object

fix: Add strong regex checking for model quality dashboard

15b5cdc

feature: add enable_automatic_dashboard and dashboard_name parameter …

812228b

…to update_monitoring_schedule

fix: Fix circular dependecy with EndpointInput and dashboards

04c912e

fix: temporary fix to allow unit tests to pass

1fb0692

chore: fix style issues

6854606

test: add unit tests for dashboard generation and error handling

b531ccf

feature: add enable_automatic_dashboard and dashboard_name parameter …

c2c9334

…to update_monitoring_schedule methods for Data Quality and Model Quality

chore: code refatoring

90c400f

fix: modify the mock boto client so that it correctly raises a Client…

7c65edf

…Error

tests: Add integration tests to publish dashboard, TODO: Verify dashb…

69e8eae

…oards are published using boto3

fix: Added additional filtering to dashboard variable in data quality…

7737941

… so that only features from the specific monitoring schedule and endpoint are loaded

fix: Fix source code to properly filter using dashboard variables and…

34d21dc

… modify test_dashboard_methods to take into account the new filters in the dashboard variables

fix: Fix docstyle check related errors

efde767

fix: Fix trailing whitespace mistakes

efbeab3

fix: Fix all style checks

827427a

sushanthkumar2004 force-pushed the data-quality-automatic-dashboard branch from e7286e6 to 827427a Compare August 23, 2024 06:31

sushanthkumar2004 requested a deployment to manual-approval August 23, 2024 06:31 — with GitHub Actions Waiting

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Data quality automatic dashboard #4785

Data quality automatic dashboard #4785

sushanthkumar2004 commented Jul 15, 2024 •

edited

Loading

zicanl-amazon Jul 15, 2024

sushanthkumar2004 Jul 16, 2024

zicanl-amazon Jul 15, 2024

sushanthkumar2004 Jul 23, 2024

sage-maker commented Aug 7, 2024

Data quality automatic dashboard #4785

Are you sure you want to change the base?

Data quality automatic dashboard #4785

Conversation

sushanthkumar2004 commented Jul 15, 2024 • edited Loading

Merge Checklist

General

Tests

zicanl-amazon Jul 15, 2024

Choose a reason for hiding this comment

sushanthkumar2004 Jul 16, 2024

Choose a reason for hiding this comment

zicanl-amazon Jul 15, 2024

Choose a reason for hiding this comment

sushanthkumar2004 Jul 23, 2024

Choose a reason for hiding this comment

sage-maker commented Aug 7, 2024

sushanthkumar2004 commented Jul 15, 2024 •

edited

Loading