Skip to content

Commit

Permalink
[Enhancement](compaction)Optimize compaction task permit allocation (#…
Browse files Browse the repository at this point in the history
…45197)

The current implementation of compaction task submission reserves
permits before task execution, which can lead to inefficient resource
utilization. Tasks waiting in the thread pool queue may hold permits,
potentially blocking other tasks from being executed.

## Solution

Change total_permits_for_compaction_score to 1,000,000, which will
effectively remove the limit on total permits. The original purpose of
total permits was to control the memory of compaction tasks, but
currently, memory is controlled by individual compaction tasks, so total
permits are no longer serving any purpose. If no memory issues arise in
the next two versions after making this change, we will remove the
permits mechanism.
  • Loading branch information
Yukang-Lian authored Dec 10, 2024
1 parent 5e655d4 commit 82d799a
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion be/src/common/config.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -455,7 +455,7 @@ DEFINE_mInt32(cumulative_compaction_max_deltas_factor, "10");
DEFINE_mInt32(multi_get_max_threads, "10");

// The upper limit of "permits" held by all compaction tasks. This config can be set to limit memory consumption for compaction.
DEFINE_mInt64(total_permits_for_compaction_score, "10000");
DEFINE_mInt64(total_permits_for_compaction_score, "1000000");

// sleep interval in ms after generated compaction tasks
DEFINE_mInt32(generate_compaction_tasks_interval_ms, "100");
Expand Down

0 comments on commit 82d799a

Please sign in to comment.