Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEAT: online anomaly detection #546

Open
wants to merge 52 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
61658c4
Handle new anomaly detection method
marcopeix Nov 19, 2024
b51e569
Merge branch 'main' into feature/anomaly_detection_v2
marcopeix Nov 21, 2024
6940e2a
WIP - Add method for online anomaly detection
marcopeix Nov 25, 2024
372b654
Merge branch 'main' into feature/anomaly_detection_v2
marcopeix Nov 25, 2024
7a52958
WIP - New method for new anomaly detection endpoint
marcopeix Nov 25, 2024
8da4495
WIP - Process output of anomaly v2 to account step size
marcopeix Nov 26, 2024
38c993d
WIP - Distributed real time anomaly detection
marcopeix Nov 26, 2024
c509c58
WIP - Write tests
marcopeix Nov 26, 2024
a0f3dfa
Merge branch 'main' into feature/anomaly_detection_v2
marcopeix Nov 26, 2024
e3fa7ce
change column names
Yibei990826 Nov 27, 2024
350ef80
include accumulated anomaly score column in output
Yibei990826 Nov 28, 2024
2de28b8
WIP - Accumulated z-scores with num_partitions
marcopeix Nov 28, 2024
9b3c2ac
raise error when num_partitions > 1 for multivariate detection
marcopeix Nov 28, 2024
12af589
Fix capabilities nb as weight_x is not accessible for anomaly detection
marcopeix Nov 28, 2024
aa022d2
Remove the use of weights_x for anomaly detection in capabilities
marcopeix Nov 28, 2024
6855602
Remove weights_x for anomaly detection in tutorials
marcopeix Nov 28, 2024
eea8bfb
Fix conflicts
marcopeix Dec 3, 2024
6bafcfe
WIP - Support standard freq and add refit to anomaly detection
marcopeix Dec 3, 2024
67e482b
Merge branch 'main' into feature/anomaly_detection_v2
marcopeix Dec 5, 2024
44bcd37
Use FreqType and tests
marcopeix Dec 5, 2024
25c24e3
remove accumulated score from schema
marcopeix Dec 5, 2024
dfe7644
Fix linting
marcopeix Dec 5, 2024
e0fcce8
remove unecessary groupby when assembling results
marcopeix Dec 9, 2024
d293408
Merge branch 'main' into feature/anomaly_detection_v2
marcopeix Dec 9, 2024
555c7d4
add tutorials for new endpoint
Yibei990826 Dec 10, 2024
5f7d614
changes with Marco' comments
Yibei990826 Dec 10, 2024
321797f
add sdk reference
Yibei990826 Dec 10, 2024
d8572c5
change tutorials
Yibei990826 Dec 11, 2024
41a7317
minor changes in formatting
Yibei990826 Dec 12, 2024
ea9eb9a
change dataset loadng path
Yibei990826 Dec 13, 2024
f0b4d09
Merge branch 'main' into feature/anomaly_detection_v2
marcopeix Dec 13, 2024
68767af
Jose comments
marcopeix Dec 13, 2024
f997f42
Fix type hinting
marcopeix Dec 13, 2024
0d91509
Another type hinting fix
marcopeix Dec 13, 2024
5247808
Adjust _process_exog_features function
marcopeix Dec 13, 2024
722d3e3
change endpoint name
Yibei990826 Dec 13, 2024
1f055cf
Remove validate_api_key and adjust tutorials
marcopeix Dec 16, 2024
f681749
remove unnecessary import in SDk reference
marcopeix Dec 16, 2024
7e40d4d
Add weights_x back to historical anomaly detection capabilities
marcopeix Dec 16, 2024
73de973
Merge branch 'main' into feature/anomaly_detection_v2
marcopeix Dec 16, 2024
1619447
Enforce hierchicalforecast<1.0.0
marcopeix Dec 16, 2024
37a3eee
Merge branch 'main' into feature/anomaly_detection_v2
marcopeix Dec 17, 2024
cb74940
remove version condition on hierarchicalforecast
marcopeix Dec 17, 2024
b4b768c
remove datasets from assets
marcopeix Dec 17, 2024
1f8c006
Fix conflicts
marcopeix Dec 18, 2024
4291b2a
type hint functions
marcopeix Dec 18, 2024
bdaa4ba
Merge branch 'main' into feature/anomaly_detection_v2
marcopeix Dec 18, 2024
a3fead2
Merge branch 'main' into feature/anomaly_detection_v2
marcopeix Dec 18, 2024
b1f3465
Merge branch 'main' into feature/anomaly_detection_v2
marcopeix Dec 20, 2024
0d78157
Change google colab url
marcopeix Dec 20, 2024
0dcdc45
Adjust online anomaly detection tutorials
marcopeix Dec 20, 2024
080df2d
Merge branch 'main' into feature/anomaly_detection_v2
Yibei990826 Dec 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36,101 changes: 36,101 additions & 0 deletions nbs/assets/SMD_test.csv

Large diffs are not rendered by default.

2,777 changes: 2,777 additions & 0 deletions nbs/assets/machine-1-1.csv

Large diffs are not rendered by default.

49 changes: 0 additions & 49 deletions nbs/docs/capabilities/anomaly-detection/00_anomaly_detection.ipynb

This file was deleted.

220 changes: 0 additions & 220 deletions nbs/docs/capabilities/anomaly-detection/02_anomaly_exogenous.ipynb

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "6de758ee-a0d2-4b3f-acff-eed419dd17c5",
"metadata": {},
"source": [
"# Historical anomaly detection"
]
},
{
"cell_type": "markdown",
"id": "5d267032-535b-4b7b-b7d3-d2db8f673af6",
"metadata": {},
"source": [
"This section provides various recipes for performing historical anomaly detection using TimeGPT.\n",
"\n",
"Historical anomaly detection identifies data points that deviate from the expected behavior over a given historical time series, helping to spot fraudulent activity, security breaches, or significant outliers.\n",
"\n",
"The process involves generating predictions and constructing a 99% confidence interval. Data points falling outside this interval are considered anomalies.\n",
"\n",
"This section covers:\n",
"\n",
"* [Historical anomaly detection](https://docs.nixtla.io/docs/capabilities-historical-anomaly-detection-quickstart)\n",
"\n",
"* [Historical anomaly detection with exogenous features](https://docs.nixtla.io/docs/capabilities-historical-anomaly-detection-add_exogenous_variables)\n",
"\n",
"* [Historical anomaly detection with date features](https://docs.nixtla.io/docs/capabilities-historical-anomaly-detection-add_date_features)\n",
"\n",
"* [Modifying the confidence intervals](https://docs.nixtla.io/docs/capabilities-historical-anomaly-detection-add_confidence_levels)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "python3",
"language": "python",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@
"source": [
"# Quickstart\n",
"\n",
"To perform anomaly detection, use the `detect_anomalies` method. Then, plot the anomalies using the `plot` method."
"To perform historical anomaly detection, use the `detect_anomalies` method. Then, plot the anomalies using the `plot` method."
]
},
{
Expand Down Expand Up @@ -176,7 +176,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"For an in-depth guide on anomaly detection with TimeGPT, check out our [tutorial](https://docs.nixtla.io/docs/tutorials-anomaly_detection)."
"For an in-depth guide on historical anomaly detection with TimeGPT, check out our [tutorial](https://docs.nixtla.io/docs/tutorials-anomaly_detection)."
]
}
],
Expand Down

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@
"source": [
"# Add confidence levels\n",
"\n",
"Tweak the confidence level used for anomaly detection. By default, if a value falls outside the 99% confidence interval, it is labeled as an anomaly.\n",
"Tweak the confidence level used for historical anomaly detection. By default, if a value falls outside the 99% confidence interval, it is labeled as an anomaly.\n",
"\n",
"Modify this with the `level` parameter, which accepts any value between 0 and 100, including decimals.\n",
"\n",
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Realtime Anomaly Detection"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Real-time anomaly detection dynamically identifies anomalies as data streams in, allowing users to specify the number of steps to monitor. This method is well-suited for immediate applications, such as fraud detection, live sensor monitoring, or tracking real-time demand changes. By focusing on recent data and continuously generating forecasts, it enables timely responses to anomalies in critical scenarios.\n",
"\n",
"This section provides various recipes for performing real-time anomaly detection using TimeGPT, offering users the ability to detect outliers and unusual patterns as they emerge, ensuring prompt intervention in time-sensitive situations.\n",
"\n",
"This section covers:\n",
"\n",
"* [Realtime anomaly detection](https://docs.nixtla.io/docs/capabilities-realtime-anomaly-detection-quickstart)\n",
"\n",
"* [How to improve the detection process](https://docs.nixtla.io/docs/capabilities-anomaly-detection-improve_detection_accuracy)\n",
"\n",
"* [Univariate vs. multiseries anomaly detection](https://docs.nixtla.io/docs/capabilities-univariate_vs_multivariate_anomaly_detection)\n"
]
}
],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 2
}
393 changes: 393 additions & 0 deletions nbs/docs/capabilities/realtime-anomaly-detection/01_quickstart.ipynb

Large diffs are not rendered by default.

Loading
Loading