Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Data Prepper opensearch sink documentation #6386

Merged
merged 12 commits into from
Mar 21, 2024

Conversation

graytaylor0
Copy link
Member

@graytaylor0 graytaylor0 commented Feb 9, 2024

Description

  • Adds missing parameters to the OpenSearch sink
  • Remove outdated parameters from OpenSearch sink
  • Add example for using actions and template_content

Resolves #4771

Checklist

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
    For more information on following Developer Certificate of Origin and signing off your commits, please check here.

number_of_replicas | No | Integer | The number of replica shards each primary shard should have on the destination OpenSearch server. For example, if you have 4 primary shards and set number_of_replicas to 3, the index has 12 replica shards. This parameter is effective only when `template_file` is either explicitly provided in Sink configuration or built-in. If this parameter is set, it would override the value in index template file. For more information, see [Create index]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/).
distribution_version | No | String | Indicates whether the sink backend version is Elasticsearch 6 or later. `es6` represents Elasticsearch 6. `default` represents the latest compatible backend version, such as Elasticsearch 7.x, OpenSearch 1.x, or OpenSearch 2.x. Default is `default`.
enable_request_compression | No | Boolean | Whether to enable compression when sending requests to OpenSearch. When `distribution_version` is set to `es6`, default is `false`. For all other distribution versions, default is `true`.
serverless | No | Boolean | Determines whether the OpenSearch backend is Amazon OpenSearch Serverless. Set this value to `true` when the destination for the `opensearch` sink is an Amazon OpenSearch Serverless collection. Default is `false`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should keep these values and note that they are deprecated in favor of the aws options.

  • serverless
  • serverless_options

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are included in the aws section already on lines 101 under the AWS section, and they appeared in the same section before. Are the settings deprecated?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These settings are deprecated. But, they are still available in the root part. So we should keep them documented here. We can clearly state they are deprecated and tell users to use the other configuration.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify, use the options under the AWS configuration and not the root, but keep the two serverless options under the root configurations stating that they are deprecated?

Signed-off-by: Naarcha-AWS <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
@Naarcha-AWS Naarcha-AWS added 5 - Editorial review PR: Editorial review in progress and removed 4 - Doc review PR: Doc review in progress labels Mar 14, 2024
Signed-off-by: Naarcha-AWS <[email protected]>
Copy link
Collaborator

@natebower natebower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@graytaylor0 @Naarcha-AWS Please see my comments and changes and let me know if you have any questions. Thanks!

_data-prepper/pipelines/configuration/sinks/opensearch.md Outdated Show resolved Hide resolved
<!-- vale off -->
Option | Required | Type | Description
:--- | :--- |:---| :---
`hosts` | Yes | List | List of OpenSearch hosts to write to such as `["https://localhost:9200", "https://remote-cluster:9200"]`.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either "A list" or "The list".

_data-prepper/pipelines/configuration/sinks/opensearch.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/configuration/sinks/opensearch.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/configuration/sinks/opensearch.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/configuration/sinks/opensearch.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/configuration/sinks/opensearch.md Outdated Show resolved Hide resolved
_data-prepper/pipelines/configuration/sinks/opensearch.md Outdated Show resolved Hide resolved
vpce_id | Yes | String | The virtual private cloud (VPC) endpoint to which the source connects.
`network_policy_name` | Yes | String | The name of the network policy to create.
`collection_name` | Yes | String | The name of the Amazon OpenSearch Serverless collection to configure.
`vpce_id` | Yes | String | The virtual private cloud (VPC) endpoint to which the source connects.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is referring to a generic VPC, then it is correct as written. If it is referring to Amazon VPC, then it should be "The Amazon Virtual Private Cloud (Amazon VPC) endpoint".

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generic. Theoretically, someone couple use AWS with their own non-AWS VPC.

_data-prepper/pipelines/configuration/sinks/opensearch.md Outdated Show resolved Hide resolved
Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
@Naarcha-AWS
Copy link
Collaborator

@dlvenable: Any changes need for this one before we merge?

@Naarcha-AWS Naarcha-AWS merged commit 5ca0629 into opensearch-project:data-prepper-2.7 Mar 21, 2024
3 checks passed
Naarcha-AWS added a commit that referenced this pull request Mar 27, 2024
* Obfuscate processor doc (#6387)

Signed-off-by: shaavanga <[email protected]>

* [Data Prepper] MAINT: document on disable_refresh secret extension setting (#6384)

* MAINT: document on disable_refresh secret extension setting

Signed-off-by: George Chen <[email protected]>

* Update _data-prepper/managing-data-prepper/configuring-data-prepper.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update _data-prepper/managing-data-prepper/configuring-data-prepper.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: George Chen <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>

* Updates to the S3 source documentation (#6379)

* Updates to the S3 source documentation to include missing features and metrics.

Signed-off-by: David Venable <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Try to fix build error

* Update s3.md

* Update s3.md

* Update s3.md

* Update s3.md

* Update s3.md

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update s3.md

* See if removing links removes build error

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: David Venable <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Update list_to_map processor in Data Prepper (#6382)

* Update list-to-map processor

Signed-off-by: Hai Yan <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Hai Yan <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Improvements to the S3 sink documenation (#6383)

* Corrections and clarifications on the S3 sink. Include an IAM policy and an example Parquet schema.

Signed-off-by: David Venable <[email protected]>

* Updates to the S3 sink to clarify how the object name is generated. Removes an option which does not exist.

Signed-off-by: David Venable <[email protected]>

* Update s3.md

* Update s3.md

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update _data-prepper/pipelines/configuration/sinks/s3.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update s3.md

* Add David's link

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: David Venable <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Add permissions, metrics, and metadata attributes to Data Prepper dyn… (#6380)

* Add permissions, metrics, and metadata attributes to Data Prepper dynamodb source documentation

Signed-off-by: Taylor Gray <[email protected]>

* Address PR comments

Signed-off-by: Taylor Gray <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update dynamo-db.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Date Processor doc update (#6381)

* Date Processor doc update

Signed-off-by: Asif Sohail Mohammed <[email protected]>

* Fixed table header indentation

Signed-off-by: Asif Sohail Mohammed <[email protected]>

* Fix formatting and grammar.

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update date.md

---------

Signed-off-by: Asif Sohail Mohammed <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Add docs for join function in Data Prepper (#6688)

* Add docs for join function

Signed-off-by: Hai Yan <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update expression-syntax.md

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Hai Yan <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Add docs for flatten processor for Data Prepper (#6685)

* Add docs for flatten processor

Signed-off-by: Hai Yan <[email protected]>

* Add grammar edits

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update flatten.md

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Hai Yan <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Adds data prepper decompress processor documentation (#6683)

* Add data prepper decompress processor documentation

Signed-off-by: Taylor Gray <[email protected]>

* Update decompress.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update decompress.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Add data prepper documentation for grok performance_metadata (#6681)

* Add data prepper grok performance metadata documentation

Signed-off-by: Taylor Gray <[email protected]>

* Update grok.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update grok.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Documentation for data prepper dynamodb source view_on_remove feature (#6738)

* Add documentation for dynamodb source view_on_remove feature

Signed-off-by: Taylor Gray <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>

* Update Data Prepper opensearch sink documentation (#6386)

* Update Data Prepper opensearch sink documentation

Signed-off-by: Taylor Gray <[email protected]>

* Formatting fixes and adding introductory text.

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update opensearch.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update opensearch.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update opensearch.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update opensearch.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update opensearch.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Adds a configuration for the Data Prepper S3 source workers field. (#6774)

* Adds a configuration for the Data Prepper S3 source workers field.

Signed-off-by: David Venable <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: David Venable <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>

* Add parse_ion processor (#6761)

* Add parse_ion processor

Signed-off-by: Archer <[email protected]>

* Apply suggestions from code review

Co-authored-by: David Venable <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update parse-ion.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Heather Halter <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Heather Halter <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Archer <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: David Venable <[email protected]>
Co-authored-by: Heather Halter <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Add docs for map_to_list processor (#6680)

* Add docs for map_to_list processor

Signed-off-by: Hai Yan <[email protected]>

* Update map-to-list.md

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Hai Yan <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Adds documentation for the Data Prepper geoip processor and geoip_service extension (#6772)

* Adds documentation for the Data Prepper geoip processor and geoip_service extension.

Signed-off-by: David Venable <[email protected]>

* Update extensions.md

* Update geoip_service.md

* Update geoip.md

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update _data-prepper/managing-data-prepper/extensions/extensions.md

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: David Venable <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Update flatten.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Fix nav order

Signed-off-by: Archer <[email protected]>

---------

Signed-off-by: shaavanga <[email protected]>
Signed-off-by: George Chen <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Signed-off-by: David Venable <[email protected]>
Signed-off-by: Hai Yan <[email protected]>
Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Asif Sohail Mohammed <[email protected]>
Signed-off-by: Archer <[email protected]>
Co-authored-by: Prathyusha Vangala <[email protected]>
Co-authored-by: Qi Chen <[email protected]>
Co-authored-by: David Venable <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>
Co-authored-by: Hai Yan <[email protected]>
Co-authored-by: Taylor Gray <[email protected]>
Co-authored-by: Asif Sohail Mohammed <[email protected]>
Co-authored-by: Heather Halter <[email protected]>
Naarcha-AWS added a commit that referenced this pull request Mar 27, 2024
* Obfuscate processor doc (#6387)

Signed-off-by: shaavanga <[email protected]>

* [Data Prepper] MAINT: document on disable_refresh secret extension setting (#6384)

* MAINT: document on disable_refresh secret extension setting

Signed-off-by: George Chen <[email protected]>

* Update _data-prepper/managing-data-prepper/configuring-data-prepper.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update _data-prepper/managing-data-prepper/configuring-data-prepper.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: George Chen <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>

* Updates to the S3 source documentation (#6379)

* Updates to the S3 source documentation to include missing features and metrics.

Signed-off-by: David Venable <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Try to fix build error

* Update s3.md

* Update s3.md

* Update s3.md

* Update s3.md

* Update s3.md

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update s3.md

* See if removing links removes build error

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: David Venable <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Update list_to_map processor in Data Prepper (#6382)

* Update list-to-map processor

Signed-off-by: Hai Yan <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Hai Yan <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Improvements to the S3 sink documenation (#6383)

* Corrections and clarifications on the S3 sink. Include an IAM policy and an example Parquet schema.

Signed-off-by: David Venable <[email protected]>

* Updates to the S3 sink to clarify how the object name is generated. Removes an option which does not exist.

Signed-off-by: David Venable <[email protected]>

* Update s3.md

* Update s3.md

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update _data-prepper/pipelines/configuration/sinks/s3.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update s3.md

* Add David's link

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: David Venable <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Add permissions, metrics, and metadata attributes to Data Prepper dyn… (#6380)

* Add permissions, metrics, and metadata attributes to Data Prepper dynamodb source documentation

Signed-off-by: Taylor Gray <[email protected]>

* Address PR comments

Signed-off-by: Taylor Gray <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update dynamo-db.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Date Processor doc update (#6381)

* Date Processor doc update

Signed-off-by: Asif Sohail Mohammed <[email protected]>

* Fixed table header indentation

Signed-off-by: Asif Sohail Mohammed <[email protected]>

* Fix formatting and grammar.

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update date.md

---------

Signed-off-by: Asif Sohail Mohammed <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Add docs for join function in Data Prepper (#6688)

* Add docs for join function

Signed-off-by: Hai Yan <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update expression-syntax.md

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Hai Yan <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Add docs for flatten processor for Data Prepper (#6685)

* Add docs for flatten processor

Signed-off-by: Hai Yan <[email protected]>

* Add grammar edits

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update flatten.md

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Hai Yan <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Adds data prepper decompress processor documentation (#6683)

* Add data prepper decompress processor documentation

Signed-off-by: Taylor Gray <[email protected]>

* Update decompress.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update decompress.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Add data prepper documentation for grok performance_metadata (#6681)

* Add data prepper grok performance metadata documentation

Signed-off-by: Taylor Gray <[email protected]>

* Update grok.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update grok.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Documentation for data prepper dynamodb source view_on_remove feature (#6738)

* Add documentation for dynamodb source view_on_remove feature

Signed-off-by: Taylor Gray <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>

* Update Data Prepper opensearch sink documentation (#6386)

* Update Data Prepper opensearch sink documentation

Signed-off-by: Taylor Gray <[email protected]>

* Formatting fixes and adding introductory text.

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update opensearch.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update opensearch.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update opensearch.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update opensearch.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update opensearch.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Adds a configuration for the Data Prepper S3 source workers field. (#6774)

* Adds a configuration for the Data Prepper S3 source workers field.

Signed-off-by: David Venable <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: David Venable <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>

* Add parse_ion processor (#6761)

* Add parse_ion processor

Signed-off-by: Archer <[email protected]>

* Apply suggestions from code review

Co-authored-by: David Venable <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update parse-ion.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Heather Halter <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Heather Halter <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Archer <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: David Venable <[email protected]>
Co-authored-by: Heather Halter <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Add docs for map_to_list processor (#6680)

* Add docs for map_to_list processor

Signed-off-by: Hai Yan <[email protected]>

* Update map-to-list.md

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Hai Yan <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Adds documentation for the Data Prepper geoip processor and geoip_service extension (#6772)

* Adds documentation for the Data Prepper geoip processor and geoip_service extension.

Signed-off-by: David Venable <[email protected]>

* Update extensions.md

* Update geoip_service.md

* Update geoip.md

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update _data-prepper/managing-data-prepper/extensions/extensions.md

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: David Venable <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Update flatten.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Fix nav order

Signed-off-by: Archer <[email protected]>

---------

Signed-off-by: shaavanga <[email protected]>
Signed-off-by: George Chen <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Signed-off-by: David Venable <[email protected]>
Signed-off-by: Hai Yan <[email protected]>
Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Asif Sohail Mohammed <[email protected]>
Signed-off-by: Archer <[email protected]>
Co-authored-by: Prathyusha Vangala <[email protected]>
Co-authored-by: Qi Chen <[email protected]>
Co-authored-by: David Venable <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>
Co-authored-by: Hai Yan <[email protected]>
Co-authored-by: Taylor Gray <[email protected]>
Co-authored-by: Asif Sohail Mohammed <[email protected]>
Co-authored-by: Heather Halter <[email protected]>
(cherry picked from commit 48651b0)
Naarcha-AWS added a commit that referenced this pull request Mar 27, 2024
* Obfuscate processor doc (#6387)

Signed-off-by: shaavanga <[email protected]>

* [Data Prepper] MAINT: document on disable_refresh secret extension setting (#6384)

* MAINT: document on disable_refresh secret extension setting

Signed-off-by: George Chen <[email protected]>

* Update _data-prepper/managing-data-prepper/configuring-data-prepper.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update _data-prepper/managing-data-prepper/configuring-data-prepper.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: George Chen <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>

* Updates to the S3 source documentation (#6379)

* Updates to the S3 source documentation to include missing features and metrics.

Signed-off-by: David Venable <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Try to fix build error

* Update s3.md

* Update s3.md

* Update s3.md

* Update s3.md

* Update s3.md

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update s3.md

* See if removing links removes build error

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: David Venable <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Update list_to_map processor in Data Prepper (#6382)

* Update list-to-map processor

Signed-off-by: Hai Yan <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Hai Yan <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Improvements to the S3 sink documenation (#6383)

* Corrections and clarifications on the S3 sink. Include an IAM policy and an example Parquet schema.

Signed-off-by: David Venable <[email protected]>

* Updates to the S3 sink to clarify how the object name is generated. Removes an option which does not exist.

Signed-off-by: David Venable <[email protected]>

* Update s3.md

* Update s3.md

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update _data-prepper/pipelines/configuration/sinks/s3.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update s3.md

* Add David's link

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: David Venable <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Add permissions, metrics, and metadata attributes to Data Prepper dyn… (#6380)

* Add permissions, metrics, and metadata attributes to Data Prepper dynamodb source documentation

Signed-off-by: Taylor Gray <[email protected]>

* Address PR comments

Signed-off-by: Taylor Gray <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update dynamo-db.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Date Processor doc update (#6381)

* Date Processor doc update

Signed-off-by: Asif Sohail Mohammed <[email protected]>

* Fixed table header indentation

Signed-off-by: Asif Sohail Mohammed <[email protected]>

* Fix formatting and grammar.

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update date.md

---------

Signed-off-by: Asif Sohail Mohammed <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Add docs for join function in Data Prepper (#6688)

* Add docs for join function

Signed-off-by: Hai Yan <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update expression-syntax.md

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Hai Yan <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Add docs for flatten processor for Data Prepper (#6685)

* Add docs for flatten processor

Signed-off-by: Hai Yan <[email protected]>

* Add grammar edits

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update flatten.md

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Hai Yan <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Adds data prepper decompress processor documentation (#6683)

* Add data prepper decompress processor documentation

Signed-off-by: Taylor Gray <[email protected]>

* Update decompress.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update decompress.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Add data prepper documentation for grok performance_metadata (#6681)

* Add data prepper grok performance metadata documentation

Signed-off-by: Taylor Gray <[email protected]>

* Update grok.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update grok.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Documentation for data prepper dynamodb source view_on_remove feature (#6738)

* Add documentation for dynamodb source view_on_remove feature

Signed-off-by: Taylor Gray <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>

* Update Data Prepper opensearch sink documentation (#6386)

* Update Data Prepper opensearch sink documentation

Signed-off-by: Taylor Gray <[email protected]>

* Formatting fixes and adding introductory text.

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update opensearch.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update opensearch.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update opensearch.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update opensearch.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update opensearch.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Adds a configuration for the Data Prepper S3 source workers field. (#6774)

* Adds a configuration for the Data Prepper S3 source workers field.

Signed-off-by: David Venable <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: David Venable <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>

* Add parse_ion processor (#6761)

* Add parse_ion processor

Signed-off-by: Archer <[email protected]>

* Apply suggestions from code review

Co-authored-by: David Venable <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update parse-ion.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Heather Halter <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Heather Halter <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Archer <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: David Venable <[email protected]>
Co-authored-by: Heather Halter <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Add docs for map_to_list processor (#6680)

* Add docs for map_to_list processor

Signed-off-by: Hai Yan <[email protected]>

* Update map-to-list.md

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Hai Yan <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Adds documentation for the Data Prepper geoip processor and geoip_service extension (#6772)

* Adds documentation for the Data Prepper geoip processor and geoip_service extension.

Signed-off-by: David Venable <[email protected]>

* Update extensions.md

* Update geoip_service.md

* Update geoip.md

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update _data-prepper/managing-data-prepper/extensions/extensions.md

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: David Venable <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>

* Update flatten.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Fix nav order

Signed-off-by: Archer <[email protected]>

---------

Signed-off-by: shaavanga <[email protected]>
Signed-off-by: George Chen <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Signed-off-by: David Venable <[email protected]>
Signed-off-by: Hai Yan <[email protected]>
Signed-off-by: Taylor Gray <[email protected]>
Signed-off-by: Asif Sohail Mohammed <[email protected]>
Signed-off-by: Archer <[email protected]>
Co-authored-by: Prathyusha Vangala <[email protected]>
Co-authored-by: Qi Chen <[email protected]>
Co-authored-by: David Venable <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>
Co-authored-by: Hai Yan <[email protected]>
Co-authored-by: Taylor Gray <[email protected]>
Co-authored-by: Asif Sohail Mohammed <[email protected]>
Co-authored-by: Heather Halter <[email protected]>
(cherry picked from commit 48651b0)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
5 - Editorial review PR: Editorial review in progress data-prepper 2.7
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants