Upgrade azure-eventhub input to the modern Event Hub SDK #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Switch from the v1 Outlet to the v2 PipelineClient
It seems we only need to wait for the context to be done here.
The existing input version does not handle publishing acks from the Beats pipeline. The input API v1 does not seem to offer this feature. With the transition to the input API v2, we have acks management. However, the legacy event hub SDK internally updates the checkpoint info after a successful handler call, and does not seem to offer hooks for acks management. Since the new modern SDK offers better checkpoint management, we keep the current behavior intact, and we'll implement ACKs in the event hub input v2.
Co-authored-by: Tiago Queiroz <[email protected]>
I don't wanna use `time.Sleep`, this is just a test.
Trying to listen for the ack of the last event, and store its checkpoint information
Alongside the partition ID, users can optional send event with a partition key. Add an (optional) partition key to the event hub metadata.
The new migrate_checkpoint config option controls if the input v2 should perform a migration check on start. If migrate_checkpoint is true, the input checks and performs the migration (if v1 info exists) on the very first v2 run. If migrate_checkpoint is false, the input will skip the migration assistant and will not perform any checks or migration.
Expand processor options by adding a new `start_position` configuration. Possible values for `start_position` are: - "earliest" to start from the beginning of the event hub retention period. - "latest" to start from new events. The input uses the 'start_position' option when checkpoint information from the storage account container is unavailable (on the input's first start).
Makes the receive configuration settings available for customization on the input settings. The current default values (receive_timeout: 5s, receive_count: 100) are probably fine, but it is better to make these options available to users.
zmoog
force-pushed
the
zmoog/azure-eventhub-sdk-upgrade
branch
from
9ca5a04
to
bca6e63
Compare
|
Replaced by elastic#39796 in the upstream repo. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed commit message
Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Disruptive User Impact
Author's Checklist
How to test this PR locally
Related issues
Use cases
Screenshots
Logs