Add documentation for Kinesis source in Data prepper #8252

sb2k16 · 2024-09-13T06:36:25Z

Description

This PR adds documentation for a new kinesis source in Data Prepper. It includes the following:

Example configuration
Required configuration attributes along with their description
Required permissions to run kinesis source

Issues Resolved

N/A

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-09-13T06:36:38Z

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

Signed-off-by: Souvik Bose <[email protected]>

vagimeli · 2024-09-13T16:49:44Z

@dlvenable This PR is tagged as a first-time contributor. Will you or your team provide a technical review? Once that review is done, I'll do a doc review. Thank you.

Signed-off-by: Souvik Bose <[email protected]>

dlvenable · 2024-09-19T17:10:14Z

@sb2k16 , @vagimeli , This is a new feature for Data Prepper 2.10, which has not yet been released. We should target this to a new branch. I just created one named data-prepper-2.10.

@sb2k16 , Can you edit your PR to target the data-prepper-2.10 branch?

vagimeli · 2024-09-30T20:18:11Z

@sb2k16 Is this PR ready for a doc review? I can give it a first review and then give it a final review before the Data Prepper 2.10.

dlvenable · 2024-09-30T20:18:20Z

_data-prepper/pipelines/configuration/sources/kinesis.md

+Option | Required | Type    | Description
+:--- | :--- |:--------| :---
+`max_polling_records` | No | Integer | The number of records to fetch from Kinesis during a single call to get Kinesis stream records.
+`idle_time_between_reads` | No | Integer  | The time duration to sleep in between calls to get Kinesis stream records. 


This is a Duration, not an Integer.

Thanks @dlvenable. I have made the changes.

dlvenable · 2024-09-30T20:20:17Z

_data-prepper/pipelines/configuration/sources/kinesis.md

+
+# kinesis
+
+You can use `kinesis` source in Data Prepper to ingest records from one or more [Amazon Kinesis](https://aws.amazon.com/kinesis/) Data Streams. 


Should we link here?

https://aws.amazon.com/kinesis/data-streams/

And then update the anchor to surround all of "Amazon Kinesis Data Streams?"

Thanks @dlvenable. I have made the changes.

dlvenable · 2024-09-30T20:21:49Z

_data-prepper/pipelines/configuration/sources/kinesis.md

+`buffer_timeout` | No       | Duration | The amount of time allowed for writing events to the Data Prepper buffer before timeout occurs. Any events that the source cannot write to the buffer during the specified amount of time are discarded. Default is `1s`.
+`records_to_accumulate` | No       | Integer  | The number of messages that accumulate before being written to the buffer. Default is `100`.
+`consumer_strategy` | No       | String   | Consumer strategy to use for ingesting Kinesis data streams. Default is `fan-out`. However, `polling` can also be used. if `polling` is enabled, additional configuration for `polling` will need to be added.
+`polling` | No       | String   | if `consumer_strategy` is set to `polling`, this config will need to be added. Refer to [polling](#polling).


The type here is polling. Please update the type column and have it link to the polling section below.

Thanks @dlvenable. I have made the changes.

sb2k16 · 2024-09-30T23:09:34Z

@sb2k16 , @vagimeli , This is a new feature for Data Prepper 2.10, which has not yet been released. We should target this to a new branch. I just created one named data-prepper-2.10.

@sb2k16 , Can you edit your PR to target the data-prepper-2.10 branch?

I have edited the PR to target data-prepper-2.10 branch.

Signed-off-by: Souvik Bose <[email protected]>

CONTRIBUTING.md

Signed-off-by: Melissa Vagi <[email protected]>

_analyzers/index.md

Signed-off-by: Melissa Vagi <[email protected]>

_analyzers/index.md

Signed-off-by: Melissa Vagi <[email protected]>

_analyzers/index.md

Signed-off-by: Melissa Vagi <[email protected]>

vagimeli · 2024-10-09T19:41:39Z

@sb2k16 , @vagimeli , This is a new feature for Data Prepper 2.10, which has not yet been released. We should target this to a new branch. I just created one named data-prepper-2.10.
@sb2k16 , Can you edit your PR to target the data-prepper-2.10 branch?

I have edited the PR to target data-prepper-2.10 branch.

@dlvenable Once you approve, I'll move this PR forward for publishing.

Signed-off-by: Souvik Bose <[email protected]>

dlvenable

Thank you @sb2k16 !

dlvenable · 2024-10-21T16:36:49Z

@vagimeli , Can you push this PR forward? We'd like to get this documentation out since Data Prepper 2.10 is released.

mmehrten · 2024-10-21T18:22:05Z

_data-prepper/pipelines/configuration/sources/kinesis.md

+
+The `codec` determines how the `kinesis` source parses each Amazon Kinesis Record. For increased and more efficient performance, you can use [codec combinations]({{site.url}}{{site.baseurl}}/data-prepper/common-use-cases/codec-processor-combinations/) with certain processors.
+
+### `newline` codec


Can we add JSON codec here?

Doc review complete. Edits made for clarity and style. No technical changes made. Moving to editorial. Signed-off-by: Melissa Vagi <[email protected]>

natebower

@sb2k16 @vagimeli Please see my changes and let me know if you have any questions. Thanks!

natebower · 2024-10-22T09:55:45Z

_data-prepper/pipelines/configuration/sources/kinesis.md

+
+# kinesis
+
+You can use `kinesis` source in Data Prepper to ingest records from one or more [Amazon Kinesis Data Streams](https://aws.amazon.com/kinesis/data-streams/). 


Suggested change

You can use `kinesis` source in Data Prepper to ingest records from one or more [Amazon Kinesis Data Streams](https://aws.amazon.com/kinesis/data-streams/).

You can use the Data Prepper `kinesis` source to ingest records from one or more [Amazon Kinesis Data Streams](https://aws.amazon.com/kinesis/data-streams/).

natebower · 2024-10-22T10:00:31Z

_data-prepper/pipelines/configuration/sources/kinesis.md

+Option | Required | Type | Description
+:--- |:---------| :--- | :---
+`stream_name` | Yes      | String | Defines the name of each Kinesis stream.
+`initial_position` | No       | String | Sets the `initial_position` to determine where the `kinesis` source starts reading stream records. Use `LATEST` to start from the most recent record or `EARLIEST` to start from the beginning of the stream. Default is `LATEST`.


Suggested change

`initial_position` | No | String | Sets the `initial_position` to determine where the `kinesis` source starts reading stream records. Use `LATEST` to start from the most recent record or `EARLIEST` to start from the beginning of the stream. Default is `LATEST`.

`initial_position` | No | String | Sets the `initial_position` to determine at what point the `kinesis` source starts reading stream records. Use `LATEST` to start from the most recent record or `EARLIEST` to start from the beginning of the stream. Default is `LATEST`.

natebower · 2024-10-22T10:03:02Z

_data-prepper/pipelines/configuration/sources/kinesis.md

+`stream_name` | Yes      | String | Defines the name of each Kinesis stream.
+`initial_position` | No       | String | Sets the `initial_position` to determine where the `kinesis` source starts reading stream records. Use `LATEST` to start from the most recent record or `EARLIEST` to start from the beginning of the stream. Default is `LATEST`.
+`checkpoint_interval` | No       | Duration | Configure the `checkpoint_interval` to periodically checkpoint Kinesis streams and avoid duplication of record processing. Default is `PT2M`.
+`compression` | No | String  | Specifies the compression format. To decompress records added by the [CloudWatch subscription filter](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html) to Kinesis, use the `gzip` compression format.


Suggested change

`compression` | No | String | Specifies the compression format. To decompress records added by the [CloudWatch subscription filter](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html) to Kinesis, use the `gzip` compression format.

`compression` | No | String | Specifies the compression format. To decompress records added by a [CloudWatch Logs subscription filter](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html) to Kinesis, use the `gzip` compression format.

natebower · 2024-10-22T10:04:23Z

_data-prepper/pipelines/configuration/sources/kinesis.md

+
+## codec
+
+The `codec` determines how the `kinesis` source parses each Amazon Kinesis Record. For increased and more efficient performance, you can use [codec combinations]({{site.url}}{{site.baseurl}}/data-prepper/common-use-cases/codec-processor-combinations/) with certain processors.


Suggested change

The `codec` determines how the `kinesis` source parses each Amazon Kinesis Record. For increased and more efficient performance, you can use [codec combinations]({{site.url}}{{site.baseurl}}/data-prepper/common-use-cases/codec-processor-combinations/) with certain processors.

The `codec` determines how the `kinesis` source parses each Kinesis stream record. For increased and more efficient performance, you can use [codec combinations]({{site.url}}{{site.baseurl}}/data-prepper/common-use-cases/codec-processor-combinations/) with certain processors.

natebower · 2024-10-22T10:05:31Z

_data-prepper/pipelines/configuration/sources/kinesis.md

+
+### `newline` codec
+
+The newline codec parses each Kinesis stream record as a single log event, making it ideal for processing single-line records. It also works well with the [`parse_json` processor]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/parse-json/) to parse each line.   


Suggested change

The newline codec parses each Kinesis stream record as a single log event, making it ideal for processing single-line records. It also works well with the [`parse_json` processor]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/parse-json/) to parse each line.

The `newline` codec parses each Kinesis stream record as a single log event, making it ideal for processing single-line records. It also works well with the [`parse_json` processor]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/parse-json/) to parse each line.

_data-prepper/pipelines/configuration/sources/kinesis.md

natebower · 2024-10-22T10:20:26Z

_data-prepper/pipelines/configuration/sources/kinesis.md

+
+* `recordsProcessed`: Counts the number of stream records processed from Kinesis streams.
+* `recordProcessingErrors`: Counts the number of processing errors for stream records from Kinesis streams.
+* `acknowledgementSetSuccesses`: Tracks the total number stream records processed that were successfully added to sink. 


Suggested change

* `acknowledgementSetSuccesses`: Tracks the total number stream records processed that were successfully added to sink.

* `acknowledgementSetSuccesses`: Counts the number of processed stream records that were successfully added to the sink.

_data-prepper/pipelines/configuration/sources/kinesis.md

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Souvik Bose <[email protected]>

Signed-off-by: Souvik Bose <[email protected]>

opensearch-trigger-bot · 2024-10-31T15:50:11Z

The backport to 2.17 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.17 2.17
# Navigate to the new working tree
pushd ../.worktrees/backport-2.17
# Create a new branch
git switch --create backport/backport-8252-to-2.17
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ae8e7632d4e09a4d553e51ca745d26fa4f3102e8
# Push it to GitHub
git push --set-upstream origin backport/backport-8252-to-2.17
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.17

Then, create a pull request where the base branch is 2.17 and the compare/head branch is backport/backport-8252-to-2.17.

sb2k16 requested review from kolchfa-aws, Naarcha-AWS, vagimeli, AMoo-Miki, natebower, dlvenable, stephen-crawford and epugh as code owners September 13, 2024 06:36

github-actions bot assigned kolchfa-aws Sep 13, 2024

kolchfa-aws assigned vagimeli and unassigned kolchfa-aws Sep 13, 2024

Add documentation for Kinesis source in Data prepper

36b39e3

Signed-off-by: Souvik Bose <[email protected]>

sb2k16 force-pushed the kinesis-documentation-update branch from 52890fe to 36b39e3 Compare September 13, 2024 16:11

vagimeli added 3 - Tech review PR: Tech review in progress data-prepper labels Sep 13, 2024

Add documentation for Kinesis source in Data prepper

005050f

Signed-off-by: Souvik Bose <[email protected]>

Merge branch 'main' into kinesis-documentation-update

92ef2d6

dlvenable requested changes Sep 30, 2024

View reviewed changes

sb2k16 changed the base branch from main to data-prepper-2.10 September 30, 2024 23:08

Address comments

33bba57

Signed-off-by: Souvik Bose <[email protected]>

vagimeli added 4 - Doc review PR: Doc review in progress and removed 3 - Tech review PR: Tech review in progress labels Oct 8, 2024

vagimeli reviewed Oct 8, 2024

View reviewed changes

CONTRIBUTING.md Outdated Show resolved Hide resolved

vagimeli reviewed Oct 8, 2024

View reviewed changes

CONTRIBUTING.md Outdated Show resolved Hide resolved

Update CONTRIBUTING.md

a17910c

Signed-off-by: Melissa Vagi <[email protected]>

Update CONTRIBUTING.md

77b6df1

Signed-off-by: Melissa Vagi <[email protected]>

vagimeli reviewed Oct 8, 2024

View reviewed changes

_analyzers/index.md Show resolved Hide resolved

Update _analyzers/index.md

42a2267

Signed-off-by: Melissa Vagi <[email protected]>

vagimeli reviewed Oct 8, 2024

View reviewed changes

_analyzers/index.md Show resolved Hide resolved

Update _analyzers/index.md

bf1bb3a

Signed-off-by: Melissa Vagi <[email protected]>

vagimeli reviewed Oct 8, 2024

View reviewed changes

_analyzers/index.md Outdated Show resolved Hide resolved

Update _analyzers/index.md

17c99b8

Signed-off-by: Melissa Vagi <[email protected]>

Update the documentation to include the compression config parameter

2db7171

Signed-off-by: Souvik Bose <[email protected]>

sb2k16 changed the base branch from data-prepper-2.10 to main October 21, 2024 16:26

dlvenable approved these changes Oct 21, 2024

View reviewed changes

mmehrten reviewed Oct 21, 2024

View reviewed changes

vagimeli added 2 commits October 21, 2024 16:17

Merge branch 'main' into kinesis-documentation-update

6ef1f64

Update kinesis.md

6e0ca99

Doc review complete. Edits made for clarity and style. No technical changes made. Moving to editorial. Signed-off-by: Melissa Vagi <[email protected]>

vagimeli added 5 - Editorial review PR: Editorial review in progress and removed 4 - Doc review PR: Doc review in progress labels Oct 21, 2024

vagimeli approved these changes Oct 21, 2024

View reviewed changes

natebower reviewed Oct 22, 2024

View reviewed changes

sb2k16 and others added 4 commits October 22, 2024 13:38

Update _data-prepper/pipelines/configuration/sources/kinesis.md

42d643a

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Souvik Bose <[email protected]>

Update _data-prepper/pipelines/configuration/sources/kinesis.md

d30319c

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Souvik Bose <[email protected]>

Update _data-prepper/pipelines/configuration/sources/kinesis.md

7ca6fc7

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Souvik Bose <[email protected]>

Update _data-prepper/pipelines/configuration/sources/kinesis.md

c749f2f

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Souvik Bose <[email protected]>

dlvenable approved these changes Oct 22, 2024

View reviewed changes

Address review comments.

4cb7caf

Signed-off-by: Souvik Bose <[email protected]>

sb2k16 force-pushed the kinesis-documentation-update branch from 077e12f to 4cb7caf Compare October 22, 2024 22:49

vagimeli merged commit ae8e763 into opensearch-project:main Oct 23, 2024
5 checks passed

kolchfa-aws added the backport 2.17 Backport for version 2.17 label Oct 31, 2024

kolchfa-aws mentioned this pull request Oct 31, 2024

Resolve merge conflicts #8640

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add documentation for Kinesis source in Data prepper #8252

Add documentation for Kinesis source in Data prepper #8252

sb2k16 commented Sep 13, 2024

github-actions bot commented Sep 13, 2024

vagimeli commented Sep 13, 2024

dlvenable commented Sep 19, 2024

vagimeli commented Sep 30, 2024

dlvenable Sep 30, 2024

sb2k16 Oct 1, 2024

dlvenable Sep 30, 2024

sb2k16 Oct 1, 2024

dlvenable Sep 30, 2024

sb2k16 Oct 1, 2024

sb2k16 commented Sep 30, 2024

vagimeli commented Oct 9, 2024

dlvenable left a comment

dlvenable commented Oct 21, 2024

mmehrten Oct 21, 2024

natebower left a comment

natebower Oct 22, 2024

natebower Oct 22, 2024

natebower Oct 22, 2024

natebower Oct 22, 2024

natebower Oct 22, 2024

natebower Oct 22, 2024

opensearch-trigger-bot bot commented Oct 31, 2024


		# kinesis

		You can use `kinesis` source in Data Prepper to ingest records from one or more [Amazon Kinesis](https://aws.amazon.com/kinesis/) Data Streams.


		The `codec` determines how the `kinesis` source parses each Amazon Kinesis Record. For increased and more efficient performance, you can use [codec combinations]({{site.url}}{{site.baseurl}}/data-prepper/common-use-cases/codec-processor-combinations/) with certain processors.

		### `newline` codec

	You can use `kinesis` source in Data Prepper to ingest records from one or more [Amazon Kinesis Data Streams](https://aws.amazon.com/kinesis/data-streams/).
	You can use the Data Prepper `kinesis` source to ingest records from one or more [Amazon Kinesis Data Streams](https://aws.amazon.com/kinesis/data-streams/).

	`initial_position` \| No \| String \| Sets the `initial_position` to determine where the `kinesis` source starts reading stream records. Use `LATEST` to start from the most recent record or `EARLIEST` to start from the beginning of the stream. Default is `LATEST`.
	`initial_position` \| No \| String \| Sets the `initial_position` to determine at what point the `kinesis` source starts reading stream records. Use `LATEST` to start from the most recent record or `EARLIEST` to start from the beginning of the stream. Default is `LATEST`.

	`compression` \| No \| String \| Specifies the compression format. To decompress records added by the [CloudWatch subscription filter](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html) to Kinesis, use the `gzip` compression format.
	`compression` \| No \| String \| Specifies the compression format. To decompress records added by a [CloudWatch Logs subscription filter](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html) to Kinesis, use the `gzip` compression format.


		## codec

		The `codec` determines how the `kinesis` source parses each Amazon Kinesis Record. For increased and more efficient performance, you can use [codec combinations]({{site.url}}{{site.baseurl}}/data-prepper/common-use-cases/codec-processor-combinations/) with certain processors.


		### `newline` codec

		The newline codec parses each Kinesis stream record as a single log event, making it ideal for processing single-line records. It also works well with the [`parse_json` processor]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/parse-json/) to parse each line.

	The newline codec parses each Kinesis stream record as a single log event, making it ideal for processing single-line records. It also works well with the [`parse_json` processor]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/parse-json/) to parse each line.
	The `newline` codec parses each Kinesis stream record as a single log event, making it ideal for processing single-line records. It also works well with the [`parse_json` processor]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/parse-json/) to parse each line.

	* `acknowledgementSetSuccesses`: Tracks the total number stream records processed that were successfully added to sink.
	* `acknowledgementSetSuccesses`: Counts the number of processed stream records that were successfully added to the sink.

Add documentation for Kinesis source in Data prepper #8252

Add documentation for Kinesis source in Data prepper #8252

Conversation

sb2k16 commented Sep 13, 2024

Description

Issues Resolved

Checklist

github-actions bot commented Sep 13, 2024

vagimeli commented Sep 13, 2024

dlvenable commented Sep 19, 2024

vagimeli commented Sep 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sb2k16 commented Sep 30, 2024

vagimeli commented Oct 9, 2024

dlvenable left a comment

Choose a reason for hiding this comment

dlvenable commented Oct 21, 2024

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Oct 31, 2024