Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding documentation for remote index use in AD #8191

Merged
merged 26 commits into from
Sep 13, 2024
Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
4ca8528
adding documentation for remote index use in AD
amitgalitz Sep 6, 2024
aa16e9a
Update _observing-your-data/ad/index.md
vagimeli Sep 6, 2024
351c8f0
Update _observing-your-data/ad/index.md
vagimeli Sep 6, 2024
08bc9d6
Update _observing-your-data/ad/index.md
vagimeli Sep 6, 2024
addce3b
Update _observing-your-data/ad/index.md
vagimeli Sep 6, 2024
6eac47f
adding additional security information
amitgalitz Sep 10, 2024
39bebb1
fixing formatting issues
amitgalitz Sep 10, 2024
c923eec
Update _observing-your-data/ad/index.md
vagimeli Sep 11, 2024
1672240
Update _observing-your-data/ad/index.md
vagimeli Sep 11, 2024
1586c9e
Update _observing-your-data/ad/index.md
vagimeli Sep 11, 2024
b909911
Update _observing-your-data/ad/security.md
vagimeli Sep 11, 2024
8a04f39
Update _observing-your-data/ad/security.md
vagimeli Sep 11, 2024
85ea3c8
Update _observing-your-data/ad/security.md
vagimeli Sep 11, 2024
d0f494e
Update _observing-your-data/ad/security.md
vagimeli Sep 11, 2024
bdf6b03
Update _observing-your-data/ad/security.md
vagimeli Sep 11, 2024
563a610
doc review new content and address editorial review comments
vagimeli Sep 11, 2024
7f76d59
doc review new content and address editorial review comments
vagimeli Sep 11, 2024
b278b1d
Update _observing-your-data/ad/security.md
vagimeli Sep 11, 2024
ab43ebf
Update _observing-your-data/ad/security.md
vagimeli Sep 11, 2024
52382a5
doc review new content and address editorial review comments
vagimeli Sep 11, 2024
37bbc82
Merge branch 'ad-remote-cluster' of https://github.com/amitgalitz/doc…
vagimeli Sep 11, 2024
93dc948
Update _observing-your-data/ad/security.md
vagimeli Sep 13, 2024
f89eb55
Update _observing-your-data/ad/security.md
vagimeli Sep 13, 2024
c2cf1c0
Update _observing-your-data/ad/security.md
vagimeli Sep 13, 2024
60de517
Update _observing-your-data/ad/security.md
vagimeli Sep 13, 2024
a179f64
Merge branch 'main' into ad-remote-cluster
vagimeli Sep 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions _observing-your-data/ad/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

# Anomaly detection

An anomaly in OpenSearch is any unusual behavior change in your time-series data. Anomalies can provide valuable insights into your data. For example, for IT infrastructure data, an anomaly in the memory usage metric might help you uncover early signs of a system failure.
An _anomaly_ in OpenSearch is any unusual behavior change in your time-series data. Anomalies can provide valuable insights into your data. For example, for IT infrastructure data, an anomaly in the memory usage metric might help you uncover early signs of a system failure.

It can be challenging to discover anomalies using conventional methods such as creating visualizations and dashboards. You could configure an alert based on a static threshold, but this requires prior domain knowledge and isn't adaptive to data that exhibits organic growth or seasonal behavior.

Expand All @@ -29,9 +29,14 @@
1. Add in the detector details.
- Enter a name and brief description. Make sure the name is unique and descriptive enough to help you to identify the purpose of the detector.
1. Specify the data source.
- For **Data source**, choose the index you want to use as the data source. You can optionally use index patterns to choose multiple indexes.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line below: Should "index pattern" be singular?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This singular usage sounds good to me on latest revision:
"- For Data source, choose one or more indexes to use as the data source. Alternatively, you can use an alias or index pattern to choose multiple indexes."

- For **Data source**, choose one or more indexes to use as the data source. Alternatively, you can use an alias or index pattern to choose multiple indexes.
- Detectors can use remote indexes. You can access them using the `cluster-name:index-name` pattern. See [Cross-cluster search]({{site.url}}{{site.baseurl}}/search-plugins/cross-cluster-search/) for more information. Alternatively, you can select clusters and indexes in OpenSearch Dashboards 2.17 or later. To learn about configuring remote indexes with the Security plugin enabled, see [Selecting remote indexes with fine-grained access control]({{site.url}}{{site.baseurl}}/observing-your-data/ad/security/#selecting-remote-indexes-with-fine-grained-access-control) in the [Anomaly detection security](observing-your-data/ad/security/) documentation.

Check failure on line 33 in _observing-your-data/ad/index.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.Spelling] Error: selectin. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks. Raw Output: {"message": "[OpenSearch.Spelling] Error: selectin. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_observing-your-data/ad/index.md", "range": {"start": {"line": 33, "column": 514}}}, "severity": "ERROR"}
- (Optional) For **Data filter**, filter the index you chose as the data source. From the **Data filter** menu, choose **Add data filter**, and then design your filter query by selecting **Field**, **Operator**, and **Value**, or choose **Use query DSL** and add your own JSON filter query. Only [Boolean queries]({{site.url}}{{site.baseurl}}/query-dsl/compound/bool/) are supported for query domain-specific language (DSL).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line above: Instead of "here", please use the name of the page for the link text.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@natebower This is my bad. The team added new content yesterday after the approved doc review. I needed to reread it and pull it from the editorial queue. I'll work on this and tag you one it's ready. I'm sorry.



To create a cross-cluster detector in OpenSearch Dashboards, the following [permissions]({{site.url}}{{site.baseurl}}/security/access-control/permissions/) are required: `indices:data/read/field_caps`, `indices:admin/resolve/index`, and `cluster:monitor/remote/info`.
{: .note}

#### Example filter using query DSL
The query is designed to retrieve documents in which the `urlPath.keyword` field matches one of the following specified values:

Expand Down
41 changes: 41 additions & 0 deletions _observing-your-data/ad/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,11 @@ As an admin user, you can use the Security plugin to assign specific permissions

The Security plugin has two built-in roles that cover most anomaly detection use cases: `anomaly_full_access` and `anomaly_read_access`. For descriptions of each, see [Predefined roles]({{site.url}}{{site.baseurl}}/security/access-control/users-roles#predefined-roles).

If you use OpenSearch Dashboards to create your anomaly detectors, you may experience permission issues even with `anomaly_full_access`. This issue has been resolved in OpenSearch 2.17, but for earlier versions, the following additional permissions need to be added:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"permissions issues"? Or do we actually mean "access issues"?

vagimeli marked this conversation as resolved.
Show resolved Hide resolved

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line above: What do we mean by "you may experience issues"? What "issue" may the user experience, specifically?

Copy link
Contributor

@vagimeli vagimeli Sep 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amitgalitz Please see the editor's comment above. We should give the users context about what issues they may experience. I think we're referring to permissions issues. Please confirm.

- `indices:data/read/search` -- You need this permission because the Anomaly Detection plugin needs to search the data source in order to validate whether there is enough data to train the model.
- `indices:admin/mappings/fields/get` and `indices:admin/mappings/fields/get*` -- You need these permissions to validate whether the given data source has a valid timestamp field and categorical field (in the case of creating a high-cardinality detector).

If these roles don't meet your needs, mix and match individual anomaly detection [permissions]({{site.url}}{{site.baseurl}}/security/access-control/permissions/) to suit your use case. Each action corresponds to an operation in the REST API. For example, the `cluster:admin/opensearch/ad/detector/delete` permission lets you delete detectors.

### A note on alerts and fine-grained access control
Expand All @@ -31,6 +36,42 @@ When a trigger generates an alert, the detector and monitor configurations, the

To reduce the chances of unintended users viewing metadata that could describe an index, we recommend that administrators enable role-based access control and keep these kinds of design elements in mind when assigning permissions to the intended group of users. See [Limit access by backend role](#advanced-limit-access-by-backend-role) for details.

### Selecting remote indexes with fine-grained access control

To use a remote index as a data source for a detector, see the setup steps in [Authentication flow]({{site.url}}{{site.baseurl}}/search-plugins/cross-cluster-search/#authentication-flow) in [Cross-cluster search]({{site.url}}{{site.baseurl}}/search-plugins/cross-cluster-search/). You must use a role that exists in both the remote and local clusters. The remote cluster must map the chosen role to the same username as in the local cluster.

---

#### Example: Create a new user on the local cluster

1. Create a new user on the local cluster to use for detector creation:

```
curl -XPUT -k -u 'admin:<custom-admin-password>' 'https://localhost:9200/_plugins/_security/api/internalusers/anomalyuser' -H 'Content-Type: application/json' -d '{"password":"password"}'
```
{% include copy-curl.html %}

2. Map the new user to the `anomaly_full_access` role:

```
curl -XPUT -k -u 'admin:<custom-admin-password>' -H 'Content-Type: application/json' 'https://localhost:9200/_plugins/_security/api/rolesmapping/anomaly_full_access' -d '{"users" : ["anomalyuser"]}'
```
{% include copy-curl.html %}

3. On the remote cluster, create the same user and map `anomaly_full_access` to that role:

```
curl -XPUT -k -u 'admin:<custom-admin-password>' 'https://localhost:9250/_plugins/_security/api/internalusers/anomalyuser' -H 'Content-Type: application/json' -d '{"password":"password"}'
curl -XPUT -k -u 'admin:<custom-admin-password>' -H 'Content-Type: application/json' 'https://localhost:9250/_plugins/_security/api/rolesmapping/anomaly_full_access' -d '{"users" : ["anomalyuser"]}'
```
{% include copy-curl.html %}

---

### Custom results index

To use a custom results index, you need additional permissions not included in the default roles provided by the OpenSearch Security plugin. To add these permissions, see [Step 1: Define a detector]({{site.url}}{{site.baseurl}}/observing-your-data/ad/index/#step-1-define-a-detector) in the [Anomaly detection]({{site.url}}{{site.baseurl}}/observing-your-data/ad/index/) documentation.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line above: Last sentence needs some revision.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revised

## (Advanced) Limit access by backend role

Use backend roles to configure fine-grained access to individual detectors based on roles. For example, users of different departments in an organization can view detectors owned by their own department.
Expand Down
Loading