-
Notifications
You must be signed in to change notification settings - Fork 508
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding documentation for remote index use in AD #8191
Changes from 24 commits
4ca8528
aa16e9a
351c8f0
08bc9d6
addce3b
6eac47f
39bebb1
c923eec
1672240
1586c9e
b909911
8a04f39
85ea3c8
d0f494e
bdf6b03
563a610
7f76d59
b278b1d
ab43ebf
52382a5
37bbc82
93dc948
f89eb55
c2cf1c0
60de517
a179f64
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -10,7 +10,7 @@ | |
|
||
# Anomaly detection | ||
|
||
An anomaly in OpenSearch is any unusual behavior change in your time-series data. Anomalies can provide valuable insights into your data. For example, for IT infrastructure data, an anomaly in the memory usage metric might help you uncover early signs of a system failure. | ||
An _anomaly_ in OpenSearch is any unusual behavior change in your time-series data. Anomalies can provide valuable insights into your data. For example, for IT infrastructure data, an anomaly in the memory usage metric might help you uncover early signs of a system failure. | ||
|
||
It can be challenging to discover anomalies using conventional methods such as creating visualizations and dashboards. You could configure an alert based on a static threshold, but this requires prior domain knowledge and isn't adaptive to data that exhibits organic growth or seasonal behavior. | ||
|
||
|
@@ -29,9 +29,14 @@ | |
1. Add in the detector details. | ||
- Enter a name and brief description. Make sure the name is unique and descriptive enough to help you to identify the purpose of the detector. | ||
1. Specify the data source. | ||
- For **Data source**, choose the index you want to use as the data source. You can optionally use index patterns to choose multiple indexes. | ||
- For **Data source**, choose one or more indexes to use as the data source. Alternatively, you can use an alias or index pattern to choose multiple indexes. | ||
- Detectors can use remote indexes. You can access them using the `cluster-name:index-name` pattern. See [Cross-cluster search]({{site.url}}{{site.baseurl}}/search-plugins/cross-cluster-search/) for more information. Alternatively, you can select clusters and indexes in OpenSearch Dashboards 2.17 or later. To learn about configuring remote indexes with the Security plugin enabled, see [Selecting remote indexes with fine-grained access control]({{site.url}}{{site.baseurl}}/observing-your-data/ad/security/#selecting-remote-indexes-with-fine-grained-access-control) in the [Anomaly detection security](observing-your-data/ad/security/) documentation. | ||
Check failure on line 33 in _observing-your-data/ad/index.md GitHub Actions / style-job
|
||
- (Optional) For **Data filter**, filter the index you chose as the data source. From the **Data filter** menu, choose **Add data filter**, and then design your filter query by selecting **Field**, **Operator**, and **Value**, or choose **Use query DSL** and add your own JSON filter query. Only [Boolean queries]({{site.url}}{{site.baseurl}}/query-dsl/compound/bool/) are supported for query domain-specific language (DSL). | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Line above: Instead of "here", please use the name of the page for the link text. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @natebower This is my bad. The team added new content yesterday after the approved doc review. I needed to reread it and pull it from the editorial queue. I'll work on this and tag you one it's ready. I'm sorry. |
||
|
||
|
||
To create a cross-cluster detector in OpenSearch Dashboards, the following [permissions]({{site.url}}{{site.baseurl}}/security/access-control/permissions/) are required: `indices:data/read/field_caps`, `indices:admin/resolve/index`, and `cluster:monitor/remote/info`. | ||
{: .note} | ||
|
||
#### Example filter using query DSL | ||
The query is designed to retrieve documents in which the `urlPath.keyword` field matches one of the following specified values: | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -23,6 +23,11 @@ As an admin user, you can use the Security plugin to assign specific permissions | |
|
||
The Security plugin has two built-in roles that cover most anomaly detection use cases: `anomaly_full_access` and `anomaly_read_access`. For descriptions of each, see [Predefined roles]({{site.url}}{{site.baseurl}}/security/access-control/users-roles#predefined-roles). | ||
|
||
If you use OpenSearch Dashboards to create your anomaly detectors, you may experience permission issues even with `anomaly_full_access`. This issue has been resolved in OpenSearch 2.17, but for earlier versions, the following additional permissions need to be added: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "permissions issues"? Or do we actually mean "access issues"?
vagimeli marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Line above: What do we mean by "you may experience issues"? What "issue" may the user experience, specifically? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @amitgalitz Please see the editor's comment above. We should give the users context about what issues they may experience. I think we're referring to permissions issues. Please confirm. |
||
- `indices:data/read/search` -- You need this permission because the Anomaly Detection plugin needs to search the data source in order to validate whether there is enough data to train the model. | ||
- `indices:admin/mappings/fields/get` and `indices:admin/mappings/fields/get*` -- You need these permissions to validate whether the given data source has a valid timestamp field and categorical field (in the case of creating a high-cardinality detector). | ||
|
||
If these roles don't meet your needs, mix and match individual anomaly detection [permissions]({{site.url}}{{site.baseurl}}/security/access-control/permissions/) to suit your use case. Each action corresponds to an operation in the REST API. For example, the `cluster:admin/opensearch/ad/detector/delete` permission lets you delete detectors. | ||
|
||
### A note on alerts and fine-grained access control | ||
|
@@ -31,6 +36,42 @@ When a trigger generates an alert, the detector and monitor configurations, the | |
|
||
To reduce the chances of unintended users viewing metadata that could describe an index, we recommend that administrators enable role-based access control and keep these kinds of design elements in mind when assigning permissions to the intended group of users. See [Limit access by backend role](#advanced-limit-access-by-backend-role) for details. | ||
|
||
### Selecting remote indexes with fine-grained access control | ||
|
||
To use a remote index as a data source for a detector, see the setup steps in [Authentication flow]({{site.url}}{{site.baseurl}}/search-plugins/cross-cluster-search/#authentication-flow) in [Cross-cluster search]({{site.url}}{{site.baseurl}}/search-plugins/cross-cluster-search/). You must use a role that exists in both the remote and local clusters. The remote cluster must map the chosen role to the same username as in the local cluster. | ||
|
||
--- | ||
|
||
#### Example: Create a new user on the local cluster | ||
|
||
1. Create a new user on the local cluster to use for detector creation: | ||
|
||
``` | ||
curl -XPUT -k -u 'admin:<custom-admin-password>' 'https://localhost:9200/_plugins/_security/api/internalusers/anomalyuser' -H 'Content-Type: application/json' -d '{"password":"password"}' | ||
``` | ||
{% include copy-curl.html %} | ||
|
||
2. Map the new user to the `anomaly_full_access` role: | ||
|
||
``` | ||
curl -XPUT -k -u 'admin:<custom-admin-password>' -H 'Content-Type: application/json' 'https://localhost:9200/_plugins/_security/api/rolesmapping/anomaly_full_access' -d '{"users" : ["anomalyuser"]}' | ||
``` | ||
{% include copy-curl.html %} | ||
|
||
3. On the remote cluster, create the same user and map `anomaly_full_access` to that role: | ||
|
||
``` | ||
curl -XPUT -k -u 'admin:<custom-admin-password>' 'https://localhost:9250/_plugins/_security/api/internalusers/anomalyuser' -H 'Content-Type: application/json' -d '{"password":"password"}' | ||
curl -XPUT -k -u 'admin:<custom-admin-password>' -H 'Content-Type: application/json' 'https://localhost:9250/_plugins/_security/api/rolesmapping/anomaly_full_access' -d '{"users" : ["anomalyuser"]}' | ||
``` | ||
{% include copy-curl.html %} | ||
|
||
--- | ||
|
||
### Custom results index | ||
|
||
To use a custom results index, you need additional permissions not included in the default roles provided by the OpenSearch Security plugin. To add these permissions, see [Step 1: Define a detector]({{site.url}}{{site.baseurl}}/observing-your-data/ad/index/#step-1-define-a-detector) in the [Anomaly detection]({{site.url}}{{site.baseurl}}/observing-your-data/ad/index/) documentation. | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Line above: Last sentence needs some revision. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Revised |
||
## (Advanced) Limit access by backend role | ||
|
||
Use backend roles to configure fine-grained access to individual detectors based on roles. For example, users of different departments in an organization can view detectors owned by their own department. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Line below: Should "index pattern" be singular?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This singular usage sounds good to me on latest revision:
"- For Data source, choose one or more indexes to use as the data source. Alternatively, you can use an alias or index pattern to choose multiple indexes."