From 1601859babb16c5ce7addc6b508bdb239859e829 Mon Sep 17 00:00:00 2001 From: abmdocrt Date: Tue, 10 Dec 2024 16:39:12 +0800 Subject: [PATCH] [Enhancement](compaction)Optimize compaction task permit allocation (#45197) The current implementation of compaction task submission reserves permits before task execution, which can lead to inefficient resource utilization. Tasks waiting in the thread pool queue may hold permits, potentially blocking other tasks from being executed. ## Solution Change total_permits_for_compaction_score to 1,000,000, which will effectively remove the limit on total permits. The original purpose of total permits was to control the memory of compaction tasks, but currently, memory is controlled by individual compaction tasks, so total permits are no longer serving any purpose. If no memory issues arise in the next two versions after making this change, we will remove the permits mechanism. --- be/src/common/config.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp index 9b8286518659cf..1d3b5ca395950e 100644 --- a/be/src/common/config.cpp +++ b/be/src/common/config.cpp @@ -463,7 +463,7 @@ DEFINE_mInt32(cumulative_compaction_max_deltas_factor, "10"); DEFINE_mInt32(multi_get_max_threads, "10"); // The upper limit of "permits" held by all compaction tasks. This config can be set to limit memory consumption for compaction. -DEFINE_mInt64(total_permits_for_compaction_score, "10000"); +DEFINE_mInt64(total_permits_for_compaction_score, "1000000"); // sleep interval in ms after generated compaction tasks DEFINE_mInt32(generate_compaction_tasks_interval_ms, "100");