Edit for redundant information and sections across Data Prepper #7127

vagimeli · 2024-05-09T21:25:02Z

Description

Address benchmark study findings about removing redundant information across Data Prepper docs. Consolidates information into one source doc.

Issues Resolved

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Melissa Vagi <[email protected]>

_data-prepper/pipelines/expression-syntax.md

vagimeli · 2024-05-10T19:44:20Z

@dlvenable This PR addresses the benchmark study about redundant documentation across Data Prepper. Please review the changes as soon as possible so that we can close this action item. cc: @hdhalter

Tasks completed include consolidating content, rewriting content for comprehensiveness and clarity, and deleting redundant content. The Expression syntax doc has substantial updates, including reorganizing information for better readability.

_data-prepper/index.md

Signed-off-by: Melissa Vagi <[email protected]>

Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

_data-prepper/pipelines/expression-syntax.md

Signed-off-by: Melissa Vagi <[email protected]>

_data-prepper/pipelines/expression-syntax.md

Signed-off-by: Melissa Vagi <[email protected]>

natebower

@vagimeli Please see my comments and changes and let me know if you have any questions. Thanks!

_data-prepper/index.md

natebower · 2024-07-25T13:58:07Z

_data-prepper/pipelines/pipelines.md


-Data Prepper ensures the durability and reliability of data written from sources and delivered to sinks through end-to-end (E2E) acknowledgments. An E2E acknowledgment begins at the source, which monitors a batch of events set inside pipelines and waits for a positive acknowledgment when those events are successfully pushed to sinks. When a pipeline contains multiple sinks, including sinks set as additional Data Prepper pipelines, the E2E acknowledgment sends when events are received by the final sink in a pipeline chain.
+Data Prepper ensures reliable and durable data delivery from sources to sinks through end-to-end (E2E) acknowledgments. The E2E acknowledgment process begins at the source, which monitors event batches within pipelines and awaits a positive acknowledgment upon successful delivery to sinks. In pipelines with multiple sinks, including nested Data Prepper pipelines, the E2E acknowledgment is sent when events reach the final sink in the pipeline chain. Conversely, the source sends a negative acknowledgment if an event cannot be delivered to a sink for any reason.


Suggested change

Data Prepper ensures reliable and durable data delivery from sources to sinks through end-to-end (E2E) acknowledgments. The E2E acknowledgment process begins at the source, which monitors event batches within pipelines and awaits a positive acknowledgment upon successful delivery to sinks. In pipelines with multiple sinks, including nested Data Prepper pipelines, the E2E acknowledgment is sent when events reach the final sink in the pipeline chain. Conversely, the source sends a negative acknowledgment if an event cannot be delivered to a sink for any reason.

Data Prepper ensures reliable and durable data delivery from sources to sinks through end-to-end (E2E) acknowledgments. The E2E acknowledgment process begins at the source, which monitors event batches within pipelines and waits for a positive acknowledgment upon successful delivery to the sinks. In pipelines with multiple sinks, including nested Data Prepper pipelines, the E2E acknowledgment is sent when events reach the final sink in the pipeline chain. Conversely, the source sends a negative acknowledgment if an event cannot be delivered to a sink for any reason.

natebower · 2024-07-25T13:58:37Z

_data-prepper/pipelines/pipelines.md


-Alternatively, the source sends a negative acknowledgment when an event cannot be delivered to a sink for any reason. 
+If a pipeline component fails to process and send an event, the source receives no acknowledgment. In the case of a failure, the pipeline's source times out, allowing you to take necessary actions, such as rerunning the pipeline or logging the failure.


Suggested change

If a pipeline component fails to process and send an event, the source receives no acknowledgment. In the case of a failure, the pipeline's source times out, allowing you to take necessary actions, such as rerunning the pipeline or logging the failure.

If a pipeline component fails to process and send an event, then the source receives no acknowledgment. In the case of a failure, the pipeline's source times out, allowing you to take necessary actions, such as rerunning the pipeline or logging the failure.

natebower · 2024-07-25T13:59:49Z

_data-prepper/pipelines/pipelines.md


+Pipelines also support conditional routing, which enables routing events to different sinks based on specific conditions. To add conditional routing, specify a list of named routes under the `route` component and assign specific routes to sinks using the `routes` property. Any sink with the `routes` property will only accept events matching at least one of the routing conditions.


Suggested change

Pipelines also support conditional routing, which enables routing events to different sinks based on specific conditions. To add conditional routing, specify a list of named routes under the `route` component and assign specific routes to sinks using the `routes` property. Any sink with the `routes` property will only accept events matching at least one of the routing conditions.

Pipelines also support conditional routing, which enables the routing of events to different sinks based on specific conditions. To add conditional routing, specify a list of named routes using the `route` component and assign specific routes to sinks using the `routes` property. Any sink with the `routes` property will only accept events matching at least one of the routing conditions.

natebower · 2024-07-25T14:00:50Z

_data-prepper/pipelines/pipelines.md

-Pipelines also support **conditional routing**  which allows you to route events to different sinks based on specific conditions. To add conditional routing to a pipeline, specify a list of named routes under the `route` component and add specific routes to sinks under the `routes` property. Any sink with the `routes` property will only accept events that match at least one of the routing conditions. 
-
-In the following example, `application-logs` is a named route with a condition set to `/log_type == "application"`. The route uses [Data Prepper expressions](https://github.com/opensearch-project/data-prepper/tree/main/examples) to define the conditions. Data Prepper only routes events that satisfy the condition to the first OpenSearch sink. By default, Data Prepper routes all events to a sink which does not define a route. In the example, all events route into the third OpenSearch sink.
+In the following example pipeline, `application-logs` is a named route with a condition set to `/log_type == "application"`. The route uses [Data Prepper expressions](https://github.com/opensearch-project/data-prepper/tree/main/examples) to define the conditions. Data Prepper routes events satisfying this condition to the first OpenSearch sink. By default, Data Prepper routes all events to sinks without a defined route, as shown in the third OpenSearch sink of the given pipeline.


Suggested change

In the following example pipeline, `application-logs` is a named route with a condition set to `/log_type == "application"`. The route uses [Data Prepper expressions](https://github.com/opensearch-project/data-prepper/tree/main/examples) to define the conditions. Data Prepper routes events satisfying this condition to the first OpenSearch sink. By default, Data Prepper routes all events to sinks without a defined route, as shown in the third OpenSearch sink of the given pipeline.

In the following example pipeline, `application-logs` is a named route with a condition set to `/log_type == "application"`. The route uses [Data Prepper expressions](https://github.com/opensearch-project/data-prepper/tree/main/examples) to define the condition. Data Prepper routes events satisfying this condition to the first OpenSearch sink. By default, Data Prepper routes all events to sinks without a defined route, as shown in the third OpenSearch sink of the given pipeline:

natebower · 2024-07-25T14:03:04Z

_data-prepper/pipelines/pipelines.md

-```
-
-When the same component is defined in both `pipelines.yaml` and `data-prepper-config.yaml`, the definition in the `pipelines.yaml` will overwrite the counterpart in `data-prepper-config.yaml`. For more information on shared pipeline components, see [AWS secrets extension plugin]({{site.url}}{{site.baseurl}}/data-prepper/managing-data-prepper/configuring-data-prepper/#aws-secrets-extension-plugin) for details.
+- Explore [common uses cases]({{site.url}}{{site.baseurl}}/data-prepper/common-use-cases/common-use-cases/) to view configuration examples.


Suggested change

- Explore [common uses cases]({{site.url}}{{site.baseurl}}/data-prepper/common-use-cases/common-use-cases/) to view configuration examples.

- See [Common uses cases]({{site.url}}{{site.baseurl}}/data-prepper/common-use-cases/common-use-cases/) for example configurations.

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]>

Accepted editorial suggestions. Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]>

_data-prepper/pipelines/expression-syntax.md

Accepted more editorial suggestions that were hidden. Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]>

hdhalter

@dlvenable - I'm trying to close this out for Melissa. Can you please accept my changes and address the couple of comments from Nathan? Thanks!

_data-prepper/index.md

hdhalter · 2024-07-30T20:52:58Z

_data-prepper/pipelines/configuration/buffers/buffers.md

+If not explicitly specified in the pipeline configuration, Data Prepper uses the default `bounded_blocking` buffer, which is in-memory queue bounded by the number of events it can hold. The `bounded_blocking` buffer is a convenient option when the event volume and processing rates are manageable within the available memory constraints. 
+
+Data Prepper also provides the flexibility to configure alternative buffer types based on your specific requirements. For example, <SME: What are the alternative types?>


@dlvenable - Are there alternatives besides what's listed (bounded blocking and kafka) that we can point to?

hdhalter · 2024-07-30T20:58:12Z

_data-prepper/pipelines/expression-syntax.md

+**Expression** | A generic component that contains a primary or an operator. Expressions can be nested within other expressions. An expression's imminent children can contain 0–1 operators.
+**Expression string** | The highest priority in a Data Prepper expression and supports only one expression string resulting in a return value. An expression string is not the same as an expression.
+**Literal** | A fundamental value that has no children. A literal can be one of the following: float, integer, Boolean, JSON pointer, string, or null. See [Literals](#literals).
+**Operator** | A hardcoded token that identifies the operation used in an expression.


Suggested change

**Operator** | A hardcoded token that identifies the operation used in an expression.

**Operator** | A hard-coded token that identifies the operation used in an expression.

hdhalter · 2024-07-30T21:01:49Z

_data-prepper/pipelines/expression-syntax.md

-White space is **optional** surrounding relational operators, regex equality operators, equality operators, and commas.
-White space is **required** surrounding set initializers, priority expressions, set operators, and conditional expressions.
+- **Float:** Supports values from 3.40282347 x 10^38 to 1.40239846 x 10^-45.
+- **Integer:** Supports values from -2,147,483,648 to 2,147,483,647.


Suggested change

- **Integer:** Supports values from -2,147,483,648 to 2,147,483,647.

- Integer: Supports values from -2,147,483,648 to 2,147,483,647.

hdhalter · 2024-07-30T21:02:01Z

_data-prepper/pipelines/expression-syntax.md

-White space is **required** surrounding set initializers, priority expressions, set operators, and conditional expressions.
+- **Float:** Supports values from 3.40282347 x 10^38 to 1.40239846 x 10^-45.
+- **Integer:** Supports values from -2,147,483,648 to 2,147,483,647.
+- **Boolean:** Supports `true` or `false`.


Suggested change

- **Boolean:** Supports `true` or `false`.

- Boolean: Supports `true` or `false`.

hdhalter · 2024-07-30T21:02:12Z

_data-prepper/pipelines/expression-syntax.md

+- **Float:** Supports values from 3.40282347 x 10^38 to 1.40239846 x 10^-45.
+- **Integer:** Supports values from -2,147,483,648 to 2,147,483,647.
+- **Boolean:** Supports `true` or `false`.
+- **JSON pointer:** See [JSON pointers](#json-pointers) for more information.


Suggested change

- **JSON pointer:** See [JSON pointers](#json-pointers) for more information.

- JSON pointer: See [JSON pointers](#json-pointers) for more information.

hdhalter · 2024-07-30T21:02:24Z

_data-prepper/pipelines/expression-syntax.md

+- **Integer:** Supports values from -2,147,483,648 to 2,147,483,647.
+- **Boolean:** Supports `true` or `false`.
+- **JSON pointer:** See [JSON pointers](#json-pointers) for more information.
+- **String:** Supports valid Java strings.


Suggested change

- **String:** Supports valid Java strings.

- String: Supports valid Java strings.

hdhalter · 2024-07-30T21:02:36Z

_data-prepper/pipelines/expression-syntax.md

+- **Boolean:** Supports `true` or `false`.
+- **JSON pointer:** See [JSON pointers](#json-pointers) for more information.
+- **String:** Supports valid Java strings.
+- **Null:** Supports `null` to check if a JSON pointer exists.


Suggested change

- **Null:** Supports `null` to check if a JSON pointer exists.

- Null: Supports `null` to check if a JSON pointer exists.

hdhalter · 2024-07-30T21:02:50Z

_data-prepper/pipelines/expression-syntax.md


-White space is **optional** surrounding relational operators, regex equality operators, equality operators, and commas.
-White space is **required** surrounding set initializers, priority expressions, set operators, and conditional expressions.
+- **Float:** Supports values from 3.40282347 x 10^38 to 1.40239846 x 10^-45.


hdhalter

There were a few more comments that needed to be addressed.

hdhalter · 2024-07-30T23:55:08Z

_data-prepper/pipelines/expression-syntax.md


-### JSON Pointer
-A JSON Pointer is a literal used to reference a value within an event and provided as context for an _expression string_. JSON Pointers are identified by a leading `/` containing alphanumeric characters or underscores, delimited by `/`. JSON Pointers can use an extended character set if wrapped in double quotes (`"`) using the escape character `\`. Note that JSON Pointers require `~` and `/` characters, which should be used as part of the path and not as a delimiter that needs to be escaped.
+JSON pointers can use an extended character set by wrapping the entire pointer in double quotes `""` and escaping characters using the backslash `\`. Note that the `~` and `/` characters are considered part of the pointer path and do not need to be escaped. Here are some examples of valid JSON pointers: `~0` to represent the literal character `~` or `~1` to represent the literal character `/`.


Suggested change

JSON pointers can use an extended character set by wrapping the entire pointer in double quotes `""` and escaping characters using the backslash `\`. Note that the `~` and `/` characters are considered part of the pointer path and do not need to be escaped. Here are some examples of valid JSON pointers: `~0` to represent the literal character `~` or `~1` to represent the literal character `/`.

JSON pointers can use an extended character set by wrapping the entire pointer in double quotes `""` and escaping characters using the backslash `\`. Note that the `~` and `/` characters are considered part of the pointer path and do not need to be escaped.

The following are examples of valid JSON pointers:

- `~0` to represent the literal character

- `~` or `~1` to represent the literal character `/`

hdhalter · 2024-07-30T23:56:23Z

_data-prepper/pipelines/expression-syntax.md

 ```
 <Any> and <Any>
 <Any> or <Any>
 not <Any>
 ```
+{% include copy-curl.html %}
+
+The following are some examples of using conditional expressions: 


Suggested change

The following are some examples of using conditional expressions:

The following is an example of using conditional expressions:

hdhalter · 2024-07-30T23:57:03Z

_data-prepper/pipelines/expression-syntax.md

+| `not`<br> `+`<br>  `-`| Unary logical NOT<br>Unary positive<br>Unary negative | right-to-left |
+| `<`, `<=`, `>`, `>=` | Relational operators                                  | left-to-right |
+| `==`, `!=`           | Equality operators                                    | left-to-right |
+| `and`, `or`          | Conditional expression                                | left-to-right |


hdhalter · 2024-07-30T23:58:06Z

_data-prepper/pipelines/configuration/sources/sources.md

+
+## Next steps 
+
+- Learn more about the source types listed under the Related Articles section.


I don't think it's not possible, since the Related Articles section is auto-generated.

natebower · 2024-07-31T10:25:53Z

@hdhalter Re: "I don't think it's not possible, since the Related Articles section is auto-generated", can we not link to the heading? If not, when referencing the heading, it should be in sentence case: "Related articles".

hdhalter · 2024-07-31T16:55:13Z

@hdhalter Re: "I don't think it's not possible, since the Related Articles section is auto-generated", can we not link to the heading? If not, when referencing the heading, it should be in sentence case: "Related articles".

Oops, sorry for the double negative! I don't think that section (Next steps) is necessary because it is a small topic and the Related articles section follows it. I'm going to remove it altogether.

_data-prepper/pipelines/configuration/sources/sources.md

_data-prepper/pipelines/configuration/sinks/sinks.md

_data-prepper/pipelines/configuration/processors/processors.md

_data-prepper/pipelines/configuration/buffers/buffers.md

Co-authored-by: Heather Halter <[email protected]> Signed-off-by: David Venable <[email protected]>

Signed-off-by: Heather Halter <[email protected]>

hdhalter · 2024-08-01T20:38:56Z

@natebower - all comments have been addressed and all links are fixed. Do you want to take another pass at it?

natebower · 2024-08-01T20:59:16Z

@natebower - all comments have been addressed and all links are fixed. Do you want to take another pass at it?

@hdhalter If all comments/suggestions have been addressed, then I won't need to re-review. Thanks.

* Edit for redundant information and sections across Data Prepper Signed-off-by: Melissa Vagi <[email protected]> * Edit for redundant information and sections across Data Prepper Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/index.md Signed-off-by: Melissa Vagi <[email protected]> * Update configuring-data-prepper.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/pipelines/expression-syntax.md Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/pipelines/expression-syntax.md Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/pipelines/pipelines.md Signed-off-by: Melissa Vagi <[email protected]> * Update expression-syntax.md Signed-off-by: Melissa Vagi <[email protected]> * Create Functions subpages Signed-off-by: Melissa Vagi <[email protected]> * Create functions subpages Signed-off-by: Melissa Vagi <[email protected]> * Copy edit Signed-off-by: Melissa Vagi <[email protected]> * add remaining subpages Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Apply suggestions from code review Accepted editorial suggestions. Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Apply suggestions from code review Accepted more editorial suggestions that were hidden. Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Apply suggestions from code review Co-authored-by: Heather Halter <[email protected]> Signed-off-by: David Venable <[email protected]> * removed-line Signed-off-by: Heather Halter <[email protected]> * Fixed broken link to pipelines Signed-off-by: Heather Halter <[email protected]> * Fixed broken links on Update add-entries.md Signed-off-by: Heather Halter <[email protected]> * Fixed broken link in Update dynamo-db.md Signed-off-by: Heather Halter <[email protected]> * Fixed link syntax in Update index.md Signed-off-by: Heather Halter <[email protected]> --------- Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Heather Halter <[email protected]> Signed-off-by: David Venable <[email protected]> Signed-off-by: Heather Halter <[email protected]> Co-authored-by: Heather Halter <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Co-authored-by: David Venable <[email protected]> (cherry picked from commit bc28bf9) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

… (#7889)

…search-project#7127) * Edit for redundant information and sections across Data Prepper Signed-off-by: Melissa Vagi <[email protected]> * Edit for redundant information and sections across Data Prepper Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/index.md Signed-off-by: Melissa Vagi <[email protected]> * Update configuring-data-prepper.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/pipelines/expression-syntax.md Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/pipelines/expression-syntax.md Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/pipelines/pipelines.md Signed-off-by: Melissa Vagi <[email protected]> * Update expression-syntax.md Signed-off-by: Melissa Vagi <[email protected]> * Create Functions subpages Signed-off-by: Melissa Vagi <[email protected]> * Create functions subpages Signed-off-by: Melissa Vagi <[email protected]> * Copy edit Signed-off-by: Melissa Vagi <[email protected]> * add remaining subpages Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Apply suggestions from code review Accepted editorial suggestions. Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Apply suggestions from code review Accepted more editorial suggestions that were hidden. Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Apply suggestions from code review Co-authored-by: Heather Halter <[email protected]> Signed-off-by: David Venable <[email protected]> * removed-line Signed-off-by: Heather Halter <[email protected]> * Fixed broken link to pipelines Signed-off-by: Heather Halter <[email protected]> * Fixed broken links on Update add-entries.md Signed-off-by: Heather Halter <[email protected]> * Fixed broken link in Update dynamo-db.md Signed-off-by: Heather Halter <[email protected]> * Fixed link syntax in Update index.md Signed-off-by: Heather Halter <[email protected]> --------- Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Heather Halter <[email protected]> Signed-off-by: David Venable <[email protected]> Signed-off-by: Heather Halter <[email protected]> Co-authored-by: Heather Halter <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Co-authored-by: David Venable <[email protected]>

* add ingest-attachment plugin doc Signed-off-by: Ricky Lippmann <[email protected]> * extend ingest-attachment with information how to limit content Signed-off-by: Ricky Lippmann <[email protected]> * Added target_bulk_bytes to the docs for logstash-output plugin (#7869) * Added target_bulk_bytes Signed-off-by: Sander van de Geijn <[email protected]> * Update _tools/logstash/ship-to-opensearch.md Nice Co-authored-by: Naarcha-AWS <[email protected]> Signed-off-by: Sander van de Geijn <[email protected]> * Update _tools/logstash/ship-to-opensearch.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update ship-to-opensearch.md * Remove "we" * Update ship-to-opensearch.md * Update ship-to-opensearch.md * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Sander van de Geijn <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Add doc for binary format support in k-NN (#7840) * Add doc for binary format support in k-NN Signed-off-by: Junqiu Lei <[email protected]> * Resolve tech feedback Signed-off-by: Junqiu Lei <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Add newline Signed-off-by: Fanit Kolchina <[email protected]> * Formatting Signed-off-by: Fanit Kolchina <[email protected]> * Link fix Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Add query results to examples Signed-off-by: Junqiu Lei <[email protected]> * Rephrased sentences and changed vector field name Signed-off-by: Fanit Kolchina <[email protected]> * Editorial review Signed-off-by: Fanit Kolchina <[email protected]> * Remove details from one of the requests Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: Junqiu Lei <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Edit for redundant information and sections across Data Prepper (#7127) * Edit for redundant information and sections across Data Prepper Signed-off-by: Melissa Vagi <[email protected]> * Edit for redundant information and sections across Data Prepper Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/index.md Signed-off-by: Melissa Vagi <[email protected]> * Update configuring-data-prepper.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/pipelines/expression-syntax.md Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/pipelines/expression-syntax.md Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/pipelines/pipelines.md Signed-off-by: Melissa Vagi <[email protected]> * Update expression-syntax.md Signed-off-by: Melissa Vagi <[email protected]> * Create Functions subpages Signed-off-by: Melissa Vagi <[email protected]> * Create functions subpages Signed-off-by: Melissa Vagi <[email protected]> * Copy edit Signed-off-by: Melissa Vagi <[email protected]> * add remaining subpages Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Apply suggestions from code review Accepted editorial suggestions. Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Apply suggestions from code review Accepted more editorial suggestions that were hidden. Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Apply suggestions from code review Co-authored-by: Heather Halter <[email protected]> Signed-off-by: David Venable <[email protected]> * removed-line Signed-off-by: Heather Halter <[email protected]> * Fixed broken link to pipelines Signed-off-by: Heather Halter <[email protected]> * Fixed broken links on Update add-entries.md Signed-off-by: Heather Halter <[email protected]> * Fixed broken link in Update dynamo-db.md Signed-off-by: Heather Halter <[email protected]> * Fixed link syntax in Update index.md Signed-off-by: Heather Halter <[email protected]> --------- Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Heather Halter <[email protected]> Signed-off-by: David Venable <[email protected]> Signed-off-by: Heather Halter <[email protected]> Co-authored-by: Heather Halter <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Co-authored-by: David Venable <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Update index.md (#7893) fixed typo Signed-off-by: Philipp Dünnebeil <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Fix typo and make left nav heading uniform for neural sparse processor (#7895) Signed-off-by: kolchfa-aws <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Add custom JSON lexer and highlighting color scheme (#7892) * Add custom JSON lexer and highlighting color scheme Signed-off-by: Fanit Kolchina <[email protected]> * Update _getting-started/quickstart.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> --------- Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Add model names to Vale (#7901) Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Renamed data prepper files to have dashes for consistency (#7790) * Renamed data prepper files to have dashes for consistency Signed-off-by: Fanit Kolchina <[email protected]> * More files Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Add documentation for ml inference search request processor/ search response processor (#7852) * draft ml inference search request processor Signed-off-by: Mingshi Liu <[email protected]> * add doc Signed-off-by: Mingshi Liu <[email protected]> * add doc Signed-off-by: Mingshi Liu <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Fixed links Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Unify processor docs Signed-off-by: Fanit Kolchina <[email protected]> * Update _query-dsl/geo-and-xy/xy.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Remove note Signed-off-by: Fanit Kolchina <[email protected]> * Fix link Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: Mingshi Liu <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Refactor k-NN documentation (#7890) * Refactor k-NN documentation Signed-off-by: Fanit Kolchina <[email protected]> * Change field name for cohesiveness Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Heather Halter <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> --------- Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Heather Halter <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Ml commons batch inference (#7899) * add batch inference API Signed-off-by: Xun Zhang <[email protected]> * add more links and mark the api as experimental Signed-off-by: Xun Zhang <[email protected]> * use openAI as the blueprint example details Signed-off-by: Xun Zhang <[email protected]> * address comments Signed-off-by: Xun Zhang <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> --------- Signed-off-by: Xun Zhang <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Remove repeated sentence in distributed tracing doc (#7906) Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Add apostrophe token filter page #7871 (#7884) * adding apostrophe token filter page #7871 Signed-off-by: AntonEliatra <[email protected]> * fixing vale error Signed-off-by: AntonEliatra <[email protected]> * Update apostrophe-token-filter.md Signed-off-by: AntonEliatra <[email protected]> * updating the naming Signed-off-by: AntonEliatra <[email protected]> * updating as per the review comments Signed-off-by: AntonEliatra <[email protected]> * updating the heading to Apostrophe token filter Signed-off-by: AntonEliatra <[email protected]> * updating as per PR comments Signed-off-by: AntonEliatra <[email protected]> * Apply suggestions from code review Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: AntonEliatra <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: AntonEliatra <[email protected]> --------- Signed-off-by: AntonEliatra <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * removed unnecessary backslash Signed-off-by: Ricky Lippmann <[email protected]> * fix:add missing whitespace in table Signed-off-by: Ricky Lippmann <[email protected]> * docs: add link to tika supported file formats Signed-off-by: Ricky Lippmann <[email protected]> * Update ingest-attachment-plugin.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * adjust to keep technical specific information with improved wording Signed-off-by: Ricky Lippmann <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Ricky Lippmann <[email protected]> Signed-off-by: Sander van de Geijn <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Signed-off-by: Junqiu Lei <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Heather Halter <[email protected]> Signed-off-by: David Venable <[email protected]> Signed-off-by: Heather Halter <[email protected]> Signed-off-by: Philipp Dünnebeil <[email protected]> Signed-off-by: Mingshi Liu <[email protected]> Signed-off-by: Xun Zhang <[email protected]> Signed-off-by: Peter Alfonsi <[email protected]> Signed-off-by: AntonEliatra <[email protected]> Co-authored-by: Sander van de Geijn <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Co-authored-by: Junqiu Lei <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Melissa Vagi <[email protected]> Co-authored-by: Heather Halter <[email protected]> Co-authored-by: David Venable <[email protected]> Co-authored-by: Philipp Dünnebeil <[email protected]> Co-authored-by: Mingshi Liu <[email protected]> Co-authored-by: Xun Zhang <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: AntonEliatra <[email protected]>

* add ingest-attachment plugin doc Signed-off-by: Ricky Lippmann <[email protected]> * extend ingest-attachment with information how to limit content Signed-off-by: Ricky Lippmann <[email protected]> * Added target_bulk_bytes to the docs for logstash-output plugin (#7869) * Added target_bulk_bytes Signed-off-by: Sander van de Geijn <[email protected]> * Update _tools/logstash/ship-to-opensearch.md Nice Co-authored-by: Naarcha-AWS <[email protected]> Signed-off-by: Sander van de Geijn <[email protected]> * Update _tools/logstash/ship-to-opensearch.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update ship-to-opensearch.md * Remove "we" * Update ship-to-opensearch.md * Update ship-to-opensearch.md * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Sander van de Geijn <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Add doc for binary format support in k-NN (#7840) * Add doc for binary format support in k-NN Signed-off-by: Junqiu Lei <[email protected]> * Resolve tech feedback Signed-off-by: Junqiu Lei <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Add newline Signed-off-by: Fanit Kolchina <[email protected]> * Formatting Signed-off-by: Fanit Kolchina <[email protected]> * Link fix Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Add query results to examples Signed-off-by: Junqiu Lei <[email protected]> * Rephrased sentences and changed vector field name Signed-off-by: Fanit Kolchina <[email protected]> * Editorial review Signed-off-by: Fanit Kolchina <[email protected]> * Remove details from one of the requests Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: Junqiu Lei <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Edit for redundant information and sections across Data Prepper (#7127) * Edit for redundant information and sections across Data Prepper Signed-off-by: Melissa Vagi <[email protected]> * Edit for redundant information and sections across Data Prepper Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Rewrite expression syntax and reorganize doc structure for readability Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/index.md Signed-off-by: Melissa Vagi <[email protected]> * Update configuring-data-prepper.md Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/pipelines/expression-syntax.md Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/pipelines/expression-syntax.md Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/pipelines/pipelines.md Signed-off-by: Melissa Vagi <[email protected]> * Update expression-syntax.md Signed-off-by: Melissa Vagi <[email protected]> * Create Functions subpages Signed-off-by: Melissa Vagi <[email protected]> * Create functions subpages Signed-off-by: Melissa Vagi <[email protected]> * Copy edit Signed-off-by: Melissa Vagi <[email protected]> * add remaining subpages Signed-off-by: Melissa Vagi <[email protected]> * Update _data-prepper/index.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Apply suggestions from code review Accepted editorial suggestions. Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Apply suggestions from code review Accepted more editorial suggestions that were hidden. Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]> * Apply suggestions from code review Co-authored-by: Heather Halter <[email protected]> Signed-off-by: David Venable <[email protected]> * removed-line Signed-off-by: Heather Halter <[email protected]> * Fixed broken link to pipelines Signed-off-by: Heather Halter <[email protected]> * Fixed broken links on Update add-entries.md Signed-off-by: Heather Halter <[email protected]> * Fixed broken link in Update dynamo-db.md Signed-off-by: Heather Halter <[email protected]> * Fixed link syntax in Update index.md Signed-off-by: Heather Halter <[email protected]> --------- Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Heather Halter <[email protected]> Signed-off-by: David Venable <[email protected]> Signed-off-by: Heather Halter <[email protected]> Co-authored-by: Heather Halter <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Co-authored-by: David Venable <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Update index.md (#7893) fixed typo Signed-off-by: Philipp Dünnebeil <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Fix typo and make left nav heading uniform for neural sparse processor (#7895) Signed-off-by: kolchfa-aws <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Add custom JSON lexer and highlighting color scheme (#7892) * Add custom JSON lexer and highlighting color scheme Signed-off-by: Fanit Kolchina <[email protected]> * Update _getting-started/quickstart.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> --------- Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Add model names to Vale (#7901) Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Renamed data prepper files to have dashes for consistency (#7790) * Renamed data prepper files to have dashes for consistency Signed-off-by: Fanit Kolchina <[email protected]> * More files Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Add documentation for ml inference search request processor/ search response processor (#7852) * draft ml inference search request processor Signed-off-by: Mingshi Liu <[email protected]> * add doc Signed-off-by: Mingshi Liu <[email protected]> * add doc Signed-off-by: Mingshi Liu <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Fixed links Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Unify processor docs Signed-off-by: Fanit Kolchina <[email protected]> * Update _query-dsl/geo-and-xy/xy.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Remove note Signed-off-by: Fanit Kolchina <[email protected]> * Fix link Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: Mingshi Liu <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Refactor k-NN documentation (#7890) * Refactor k-NN documentation Signed-off-by: Fanit Kolchina <[email protected]> * Change field name for cohesiveness Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Heather Halter <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> --------- Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Heather Halter <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Ml commons batch inference (#7899) * add batch inference API Signed-off-by: Xun Zhang <[email protected]> * add more links and mark the api as experimental Signed-off-by: Xun Zhang <[email protected]> * use openAI as the blueprint example details Signed-off-by: Xun Zhang <[email protected]> * address comments Signed-off-by: Xun Zhang <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> --------- Signed-off-by: Xun Zhang <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Remove repeated sentence in distributed tracing doc (#7906) Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * Add apostrophe token filter page #7871 (#7884) * adding apostrophe token filter page #7871 Signed-off-by: AntonEliatra <[email protected]> * fixing vale error Signed-off-by: AntonEliatra <[email protected]> * Update apostrophe-token-filter.md Signed-off-by: AntonEliatra <[email protected]> * updating the naming Signed-off-by: AntonEliatra <[email protected]> * updating as per the review comments Signed-off-by: AntonEliatra <[email protected]> * updating the heading to Apostrophe token filter Signed-off-by: AntonEliatra <[email protected]> * updating as per PR comments Signed-off-by: AntonEliatra <[email protected]> * Apply suggestions from code review Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: AntonEliatra <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: AntonEliatra <[email protected]> --------- Signed-off-by: AntonEliatra <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Ricky Lippmann <[email protected]> * removed unnecessary backslash Signed-off-by: Ricky Lippmann <[email protected]> * fix:add missing whitespace in table Signed-off-by: Ricky Lippmann <[email protected]> * docs: add link to tika supported file formats Signed-off-by: Ricky Lippmann <[email protected]> * Update ingest-attachment-plugin.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * adjust to keep technical specific information with improved wording Signed-off-by: Ricky Lippmann <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Ricky Lippmann <[email protected]> Signed-off-by: Sander van de Geijn <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Signed-off-by: Junqiu Lei <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Heather Halter <[email protected]> Signed-off-by: David Venable <[email protected]> Signed-off-by: Heather Halter <[email protected]> Signed-off-by: Philipp Dünnebeil <[email protected]> Signed-off-by: Mingshi Liu <[email protected]> Signed-off-by: Xun Zhang <[email protected]> Signed-off-by: Peter Alfonsi <[email protected]> Signed-off-by: AntonEliatra <[email protected]> Co-authored-by: Sander van de Geijn <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Co-authored-by: Junqiu Lei <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Melissa Vagi <[email protected]> Co-authored-by: Heather Halter <[email protected]> Co-authored-by: David Venable <[email protected]> Co-authored-by: Philipp Dünnebeil <[email protected]> Co-authored-by: Mingshi Liu <[email protected]> Co-authored-by: Xun Zhang <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> Co-authored-by: AntonEliatra <[email protected]> (cherry picked from commit 8b731c5) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Edit for redundant information and sections across Data Prepper

16b2387

Signed-off-by: Melissa Vagi <[email protected]>

vagimeli requested review from hdhalter, kolchfa-aws, Naarcha-AWS, AMoo-Miki, natebower, dlvenable, stephen-crawford and epugh as code owners May 9, 2024 21:25

vagimeli added 6 commits May 9, 2024 16:31

Edit for redundant information and sections across Data Prepper

584e08d

Signed-off-by: Melissa Vagi <[email protected]>

Rewrite expression syntax and reorganize doc structure for readability

458079e

Signed-off-by: Melissa Vagi <[email protected]>

Rewrite expression syntax and reorganize doc structure for readability

92cd1b8

Signed-off-by: Melissa Vagi <[email protected]>

Rewrite expression syntax and reorganize doc structure for readability

7228bb2

Signed-off-by: Melissa Vagi <[email protected]>

Rewrite expression syntax and reorganize doc structure for readability

c2d751f

Signed-off-by: Melissa Vagi <[email protected]>

Rewrite expression syntax and reorganize doc structure for readability

82373d7

Signed-off-by: Melissa Vagi <[email protected]>

vagimeli commented May 10, 2024

View reviewed changes

_data-prepper/pipelines/expression-syntax.md Show resolved Hide resolved

Merge branch 'main' into update-index

26c2013

vagimeli commented May 10, 2024

View reviewed changes

_data-prepper/index.md Outdated Show resolved Hide resolved

vagimeli added 2 commits May 10, 2024 13:46

Update _data-prepper/index.md

050292a

Signed-off-by: Melissa Vagi <[email protected]>

Update configuring-data-prepper.md

01b54ac

Signed-off-by: Melissa Vagi <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>

vagimeli commented May 10, 2024

View reviewed changes

_data-prepper/pipelines/expression-syntax.md Outdated Show resolved Hide resolved

Update _data-prepper/pipelines/expression-syntax.md

fe74f47

Signed-off-by: Melissa Vagi <[email protected]>

vagimeli commented May 10, 2024

View reviewed changes

_data-prepper/pipelines/expression-syntax.md Outdated Show resolved Hide resolved

Update _data-prepper/pipelines/expression-syntax.md

eb38891

Signed-off-by: Melissa Vagi <[email protected]>

vagimeli assigned dlvenable and vagimeli May 10, 2024

Merge branch 'main' into update-index

1cb7fb6

hdhalter added 3 - Tech review PR: Tech review in progress backport 2.14 PR: Backport label for 2.14 labels May 15, 2024

hdhalter added the backport 2.15 label Jul 19, 2024

natebower reviewed Jul 25, 2024

View reviewed changes

hdhalter and others added 2 commits July 30, 2024 11:38

Update _data-prepper/index.md

b47b8b4

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]>

Apply suggestions from code review

c647da6

Accepted editorial suggestions. Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]>

hdhalter reviewed Jul 30, 2024

View reviewed changes

_data-prepper/pipelines/expression-syntax.md Outdated Show resolved Hide resolved

Apply suggestions from code review

365fa7d

Accepted more editorial suggestions that were hidden. Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Heather Halter <[email protected]>

hdhalter reviewed Jul 30, 2024

View reviewed changes

hdhalter reviewed Jul 31, 2024

View reviewed changes

dlvenable and others added 8 commits July 31, 2024 18:59

Apply suggestions from code review

1d735a6

Co-authored-by: Heather Halter <[email protected]> Signed-off-by: David Venable <[email protected]>

removed-line

e0f7743

Signed-off-by: Heather Halter <[email protected]>

Merge branch 'main' into update-index

1b1f1f1

Merge branch 'main' into update-index

36bf8b8

Fixed broken link to pipelines

be987b5

Signed-off-by: Heather Halter <[email protected]>

Fixed broken links on Update add-entries.md

83b4136

Signed-off-by: Heather Halter <[email protected]>

Fixed broken link in Update dynamo-db.md

6bd246f

Signed-off-by: Heather Halter <[email protected]>

Fixed link syntax in Update index.md

475e202

Signed-off-by: Heather Halter <[email protected]>

hdhalter added 3 - Done Issue is done/complete and removed 5 - Editorial review PR: Editorial review in progress labels Aug 1, 2024

hdhalter merged commit bc28bf9 into main Aug 1, 2024
7 checks passed

opensearch-trigger-bot bot mentioned this pull request Aug 1, 2024

[Backport 2.15] Edit for redundant information and sections across Data Prepper #7889

Merged

github-actions bot pushed a commit that referenced this pull request Aug 1, 2024

Edit for redundant information and sections across Data Prepper (#7127)…

654fc01

… (#7889)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Edit for redundant information and sections across Data Prepper #7127

Edit for redundant information and sections across Data Prepper #7127

vagimeli commented May 9, 2024

vagimeli commented May 10, 2024 •

edited

Loading

natebower left a comment

natebower Jul 25, 2024

natebower Jul 25, 2024

natebower Jul 25, 2024

natebower Jul 25, 2024

natebower Jul 25, 2024

hdhalter left a comment

hdhalter Jul 30, 2024

hdhalter Jul 30, 2024

hdhalter Jul 30, 2024

hdhalter Jul 30, 2024

hdhalter Jul 30, 2024

hdhalter Jul 30, 2024

hdhalter Jul 30, 2024

hdhalter Jul 30, 2024

hdhalter left a comment

hdhalter Jul 30, 2024

hdhalter Jul 30, 2024

hdhalter Jul 30, 2024

hdhalter Jul 30, 2024

natebower commented Jul 31, 2024

hdhalter commented Jul 31, 2024

hdhalter commented Aug 1, 2024

natebower commented Aug 1, 2024


		Data Prepper ensures the durability and reliability of data written from sources and delivered to sinks through end-to-end (E2E) acknowledgments. An E2E acknowledgment begins at the source, which monitors a batch of events set inside pipelines and waits for a positive acknowledgment when those events are successfully pushed to sinks. When a pipeline contains multiple sinks, including sinks set as additional Data Prepper pipelines, the E2E acknowledgment sends when events are received by the final sink in a pipeline chain.
		Data Prepper ensures reliable and durable data delivery from sources to sinks through end-to-end (E2E) acknowledgments. The E2E acknowledgment process begins at the source, which monitors event batches within pipelines and awaits a positive acknowledgment upon successful delivery to sinks. In pipelines with multiple sinks, including nested Data Prepper pipelines, the E2E acknowledgment is sent when events reach the final sink in the pipeline chain. Conversely, the source sends a negative acknowledgment if an event cannot be delivered to a sink for any reason.


		Alternatively, the source sends a negative acknowledgment when an event cannot be delivered to a sink for any reason.
		If a pipeline component fails to process and send an event, the source receives no acknowledgment. In the case of a failure, the pipeline's source times out, allowing you to take necessary actions, such as rerunning the pipeline or logging the failure.


		Pipelines also support conditional routing, which enables routing events to different sinks based on specific conditions. To add conditional routing, specify a list of named routes under the `route` component and assign specific routes to sinks using the `routes` property. Any sink with the `routes` property will only accept events matching at least one of the routing conditions.

	- Explore [common uses cases]({{site.url}}{{site.baseurl}}/data-prepper/common-use-cases/common-use-cases/) to view configuration examples.
	- See [Common uses cases]({{site.url}}{{site.baseurl}}/data-prepper/common-use-cases/common-use-cases/) for example configurations.

		If not explicitly specified in the pipeline configuration, Data Prepper uses the default `bounded_blocking` buffer, which is in-memory queue bounded by the number of events it can hold. The `bounded_blocking` buffer is a convenient option when the event volume and processing rates are manageable within the available memory constraints.

		Data Prepper also provides the flexibility to configure alternative buffer types based on your specific requirements. For example, <SME: What are the alternative types?>

	Operator \| A hardcoded token that identifies the operation used in an expression.
	Operator \| A hard-coded token that identifies the operation used in an expression.

	- Integer: Supports values from -2,147,483,648 to 2,147,483,647.
	- Integer: Supports values from -2,147,483,648 to 2,147,483,647.

	- Boolean: Supports `true` or `false`.
	- Boolean: Supports `true` or `false`.

	- JSON pointer: See [JSON pointers](#json-pointers) for more information.
	- JSON pointer: See [JSON pointers](#json-pointers) for more information.

	- String: Supports valid Java strings.
	- String: Supports valid Java strings.

	- Null: Supports `null` to check if a JSON pointer exists.
	- Null: Supports `null` to check if a JSON pointer exists.

	The following are some examples of using conditional expressions:
	The following is an example of using conditional expressions:

	\| `and`, `or` \| Conditional expression \| left-to-right \|
	\| `and`, `or` \| Conditional expressions \| left-to-right \|


		## Next steps

		- Learn more about the source types listed under the Related Articles section.

Edit for redundant information and sections across Data Prepper #7127

Edit for redundant information and sections across Data Prepper #7127

Conversation

vagimeli commented May 9, 2024

Description

Issues Resolved

Checklist

vagimeli commented May 10, 2024 • edited Loading

natebower left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hdhalter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hdhalter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

natebower commented Jul 31, 2024

hdhalter commented Jul 31, 2024

hdhalter commented Aug 1, 2024

natebower commented Aug 1, 2024

vagimeli commented May 10, 2024 •

edited

Loading