Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] Define and standardize some common configurations for all Sources #5101

Open
3 tasks done
cnzakii opened this issue Sep 8, 2024 · 0 comments · May be fixed by #5102
Open
3 tasks done

[Enhancement] Define and standardize some common configurations for all Sources #5101

cnzakii opened this issue Sep 8, 2024 · 0 comments · May be fixed by #5102
Labels
enhancement New feature or request

Comments

@cnzakii
Copy link
Contributor

cnzakii commented Sep 8, 2024

Search before asking

  • I had searched in the issues and found no similar issues.

Enhancement Request

In the various implementations of Source, there are often some common configurations scattered across different places, such as the capacity of the storage queue, the number of elements to poll each time, etc. Additionally, in some source implementations, the poll method can block for up to several seconds in the worst-case scenario, reaching maxBatchSize * pollTimeout. Therefore, it is necessary to define common parameters that support configuration and also limit the maximum waiting time for poll.

Describe the solution you'd like

  1. Added a PollConfig configuration class that includes parameters for capacity, maxBatchSize, and maxWaitTime, and incorporated it as a variable in SourceConfig to support user configuration with default values.
  2. Replaced the fixed constants for capacity, maxBatchSize, and maxWaitTime in the original source implementations with the corresponding variables from PollConfig.
  3. Optimized some poll methods to ensure the maximum waiting time does not exceed maxWaitTime, preventing blocking for several seconds.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@cnzakii cnzakii added the enhancement New feature or request label Sep 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant