Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add backpressure option for bulk import #5023

Open
ddanielr opened this issue Oct 30, 2024 · 1 comment
Open

Add backpressure option for bulk import #5023

ddanielr opened this issue Oct 30, 2024 · 1 comment
Assignees
Labels
enhancement This issue describes a new feature, improvement, or optimization.

Comments

@ddanielr
Copy link
Contributor

Is your feature request related to a problem? Please describe.
Bulk import will continue to push files into tablets to a point where user scan performance can degrade.
A user can monitor the number of queued compactions before submitting new bulk import operations.
However, this approach blocks all bulk import operations even if the data was going to tablets which would not degrade scan performance.

Describe the solution you'd like
A bulk import limit threshold property based off of the table.max.file property value should be added.
This would allow bulk import to continue importing into tablets that are not exceeding this new property value and either wait indefinitely or block the fate on those specific tablets.

The bulk import operation should then export the tablets information in order for the user or a separate process can take that input and schedule higher priority compaction jobs to unblock the bulk import operation.

@ddanielr ddanielr added the enhancement This issue describes a new feature, improvement, or optimization. label Oct 30, 2024
@dlmarion
Copy link
Contributor

In #5026 I increased the priority of major compactions where the compaction manager realizes that the tablet is over the file size threshold and there is no compaction queued up. This might alleviate the condition to some degree.

@keith-turner keith-turner self-assigned this Oct 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement This issue describes a new feature, improvement, or optimization.
Projects
None yet
Development

No branches or pull requests

3 participants