Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update expression and anomaly detector documentation #8041

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
+63 −2
Open

Update expression and anomaly detector documentation #8041

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
2178906
Update expression and anomaly detector documentation
kkondaka Aug 16, 2024
12ea363
Update _data-prepper/pipelines/expression-syntax.md
vagimeli Aug 19, 2024
60b4977
Update _data-prepper/pipelines/expression-syntax.md
vagimeli Aug 19, 2024
3fbd722
Update _data-prepper/pipelines/expression-syntax.md
vagimeli Aug 19, 2024
d442ceb
Update _data-prepper/pipelines/expression-syntax.md
vagimeli Aug 19, 2024
b0c9fc4
Update _data-prepper/pipelines/expression-syntax.md
vagimeli Aug 19, 2024
3e86e12
Merge branch 'main' into expr-doc-fix
vagimeli Aug 19, 2024
8730369
Merge branch 'main' into expr-doc-fix
vagimeli Aug 22, 2024
2d870c9
Merge branch 'main' into expr-doc-fix
vagimeli Oct 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions _data-prepper/pipelines/configuration/processors/anomaly-detector.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@
| `sample_size` | `256` | 100--2500 | The sample size used in the ML algorithm. |
| `time_decay` | `0.1` | 0--1.0 | The time decay value used in the ML algorithm. Used as the mathematical expression `timeDecay` divided by `SampleSize` in the ML algorithm. |
| `type` | `metrics` | N/A | The type of data sent to the algorithm. |
| `output_after` | 32 | N/A | Indicates the number of events to consume before outputting anamolies |

Check failure on line 56 in _data-prepper/pipelines/configuration/processors/anomaly-detector.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: anamolies. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: anamolies. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/configuration/processors/anomaly-detector.md", "range": {"start": {"line": 56, "column": 91}}}, "severity": "ERROR"}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| `output_after` | 32 | N/A | Indicates the number of events to consume before outputting anamolies |
| `output_after` | 32 | N/A | Specifies the number of events to process before outputting any detected anomalies. |

| `version` | `1.0` | N/A | The algorithm version number. |

## Usage
Expand Down
64 changes: 62 additions & 2 deletions _data-prepper/pipelines/expression-syntax.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@
|----------------------|-------------------------------------------------------|---------------|
| `()` | Priority expression | Left to right |
| `not`<br> `+`<br> `-`| Unary logical NOT<br>Unary positive<br>Unary negative | Right to left |
| `*`, `/` | Multiplication and Division operators | Left to right |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
| `+`, `-` | Addition and Subtraction operators | Left to right |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
| `+` | String concatenation operator | Left to right |
| `<`, `<=`, `>`, `>=` | Relational operators | Left to right |
| `==`, `!=` | Equality operators | Left to right |
| `and`, `or` | Conditional expression | Left to right |
Expand Down Expand Up @@ -78,7 +81,6 @@
<Any> or <Any>
not <Any>
```
{% include copy-curl.html %}

The following are some example conditional expressions:

Expand All @@ -91,9 +93,64 @@
```
{% include copy-curl.html %}

### Arithmetic expressions

Arithmetic expressions allow you to do few basic arithmetic operations. Arithmetic expressions may be combined with conditional expressions to make more complex conditional expressions. The available arithmetic operators are `+`, `-`, `*`, and `/` for doing addition, subtraction, multiplication and division respectively. The syntax for using the arithmetic operators is as follows:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Arithmetic expressions allow you to do few basic arithmetic operations. Arithmetic expressions may be combined with conditional expressions to make more complex conditional expressions. The available arithmetic operators are `+`, `-`, `*`, and `/` for doing addition, subtraction, multiplication and division respectively. The syntax for using the arithmetic operators is as follows:
Arithmetic expressions enable basic mathematical operations like addition, subtraction, multiplication, and division. These expressions can be combined with conditional expressions to create more complex conditional statements. The available arithmetic operators are +, -, *, and /. The syntax for using the arithmetic operators is as follows:


```
<Any> + <Any>
<Any> - <Any>
<Any> * <Any>
<Any> / <Any>
```

The following are some example arithmetic expressions:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The following are some example arithmetic expressions:
The following are example arithmetic expressions:


```
/value + length(/message)
/bytes / 1024
/value1 - /value2
/TimeInSeconds * 1000
```
{% include copy-curl.html %}

The following are some example arithmetic expressions used in conditional expressions :
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The following are some example arithmetic expressions used in conditional expressions :
The following are example arithmetic expressions that can be used in conditional expressions:


```
/value + length(/message) > 200
/bytes / 1024 < 10
/value1 - /value2 != /value3 + /value4
```
{% include copy-curl.html %}

### String concatenation expressions

String concatenation expressions allow you to concatenate strings to generate new strings. The concatenated strings can be used in conditional expressions too. The syntax for using string concatenation is as follows:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
String concatenation expressions allow you to concatenate strings to generate new strings. The concatenated strings can be used in conditional expressions too. The syntax for using string concatenation is as follows:
String concatenation expressions enable you to combine strings to create new strings. These concatenated strings can also be used within conditional expressions. The syntax for using string concatenation is as follows:


```
<String Variable or String Literal> + <String Variable or String Literal>
```

The following are some example string concatenation expressions:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The following are some example string concatenation expressions:
The following are example string concatenation expressions:


```
/name + "suffix"
"prefix" + /name
"time of " + /timeInMs + " ms"
```
{% include copy-curl.html %}

The following are some example string concatenation expressions used in conditional expressions :
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The following are some example string concatenation expressions used in conditional expressions :
The following are example string concatenation expressions that can be used in conditional expressions:


```
/service + ".com" == /url
"www." + /service != /url
```
{% include copy-curl.html %}

### Reserved symbols

Reserved symbols are symbols that are not currently used in the expression syntax but are reserved for possible future functionality or extensions. Reserved symbols include `^`, `*`, `/`, `%`, `+`, `-`, `xor`, `=`, `+=`, `-=`, `*=`, `/=`, `%=`, `++`, `--`, and `${<text>}`.
Reserved symbols are symbols that are not currently used in the expression syntax but are reserved for possible future functionality or extensions. Reserved symbols include `^`, `%`, `xor`, `=`, `+=`, `-=`, `*=`, `/=`, `%=`, `++`, `--`, and `${<text>}`.
Copy link
Collaborator

@vagimeli vagimeli Aug 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Reserved symbols are symbols that are not currently used in the expression syntax but are reserved for possible future functionality or extensions. Reserved symbols include `^`, `%`, `xor`, `=`, `+=`, `-=`, `*=`, `/=`, `%=`, `++`, `--`, and `${<text>}`.
Certain symbols, such as ^, %, xor, =, +=, -=, *=, /=, %=, ++, --, and ${<text>}, are reserved for future functionality or extensions. Reserved symbols include `^`, `%`, `xor`, `=`, `+=`, `-=`, `*=`, `/=`, `%=`, `++`, `--`, and `${<text>}`.


## Syntax components

Expand Down Expand Up @@ -170,6 +227,9 @@
| `()` | Priority expression | Yes | `/a==(/b==200)`<br>`/a in ({200})` | `/status in({200})` |
| `in`, `not in` | Set operators | Yes | `/a in {200}`<br>`/a not in {400}` | `/a in{200, 202}`<br>`/a not in{400}` |
| `<`, `<=`, `>`, `>=` | Relational operators | No | `/status < 300`<br>`/status>=300` | |
| `+` | String concat operator | No | `/status_code + /message + "suffix"`

Check failure on line 230 in _data-prepper/pipelines/expression-syntax.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: concat. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: concat. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/expression-syntax.md", "range": {"start": {"line": 230, "column": 33}}}, "severity": "ERROR"}

Check failure on line 230 in _data-prepper/pipelines/expression-syntax.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: _code. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: _code. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/expression-syntax.md", "range": {"start": {"line": 230, "column": 85}}}, "severity": "ERROR"}
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
| `+`, `-` | Arithmetic Add and Subtract operators | No | `/status_code + length(/message) - 2`

Check failure on line 231 in _data-prepper/pipelines/expression-syntax.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: _code. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: _code. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/expression-syntax.md", "range": {"start": {"line": 231, "column": 85}}}, "severity": "ERROR"}
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
| `*`, `/` | Multiply and Divide operators | No | `/status_code * length(/message) / 3`

Check failure on line 232 in _data-prepper/pipelines/expression-syntax.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: _code. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: _code. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/expression-syntax.md", "range": {"start": {"line": 232, "column": 85}}}, "severity": "ERROR"}
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
| `=~`, `!~` | Regex equality operators | No | `/msg =~ "^\w*$"`<br>`/msg=~"^\w*$"` | |
| `==`, `!=` | Equality operators | No | `/status == 200`<br>`/status_code==200` | |
| `and`, `or`, `not` | Conditional operators | Yes | `/a<300 and /b>200` | `/b<300and/b>200` |
Expand Down
Loading