You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Bulk import will continue to push files into tablets to a point where user scan performance can degrade.
A user can monitor the number of queued compactions before submitting new bulk import operations.
However, this approach blocks all bulk import operations even if the data was going to tablets which would not degrade scan performance.
Describe the solution you'd like
A bulk import limit threshold property based off of the table.max.file property value should be added.
This would allow bulk import to continue importing into tablets that are not exceeding this new property value and either wait indefinitely or block the fate on those specific tablets.
The bulk import operation should then export the tablets information in order for the user or a separate process can take that input and schedule higher priority compaction jobs to unblock the bulk import operation.
The text was updated successfully, but these errors were encountered:
In #5026 I increased the priority of major compactions where the compaction manager realizes that the tablet is over the file size threshold and there is no compaction queued up. This might alleviate the condition to some degree.
Is your feature request related to a problem? Please describe.
Bulk import will continue to push files into tablets to a point where user scan performance can degrade.
A user can monitor the number of queued compactions before submitting new bulk import operations.
However, this approach blocks all bulk import operations even if the data was going to tablets which would not degrade scan performance.
Describe the solution you'd like
A bulk import limit threshold property based off of the
table.max.file
property value should be added.This would allow bulk import to continue importing into tablets that are not exceeding this new property value and either wait indefinitely or block the fate on those specific tablets.
The bulk import operation should then export the tablets information in order for the user or a separate process can take that input and schedule higher priority compaction jobs to unblock the bulk import operation.
The text was updated successfully, but these errors were encountered: