Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feat](scan) Adaptive scan concurrency #44690

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

[feat](scan) Adaptive scan concurrency #44690

wants to merge 3 commits into from

Conversation

zhiqiang-hhhh
Copy link
Contributor

@zhiqiang-hhhh zhiqiang-hhhh commented Nov 27, 2024 •
edited
Loading

What problem does this PR solve?

Implementation of adaptive scan concurrency. We submit all scanner to scheduler at once in the past. It will introduce much problems:

  1. The execution of scan task is not even between different query
  2. Memory peak of scan task
  3. When running scan task consumes all free block, other remaining scanner will be bubbles in scan scheduler.

So that we want to adjust concurrency of scanner of each scan operator by monitoring system scan pressure.

  1. Make full utilization of resource even if there is just one query.
  2. Distribute resource between all scanners.

Two new session variables are introduced:

  1. min_scanner_concurrnency: Each scan operator will have at least min_scanner_concurrnency scanner is running;
  2. min_scan_scheduler_concurrency: Minimum active threads of scan scheduler.

The original num_scanner_threads is used as max_scanner_concurrency now.

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhiqiang-hhhh zhiqiang-hhhh force-pushed the feat-concurrcy-scan branch 3 times, most recently from 0bf7245 to 990b804 Compare December 2, 2024 11:13
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

1 similar comment
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

2 similar comments
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh zhiqiang-hhhh force-pushed the feat-concurrcy-scan branch 2 times, most recently from 5b91373 to 5bc8071 Compare December 6, 2024 07:54
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Dec 10, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/util/interval_histogram.h Outdated Show resolved Hide resolved
be/src/util/interval_histogram.h Outdated Show resolved Hide resolved
be/src/util/interval_histogram.h Outdated Show resolved Hide resolved
be/src/vec/exec/scan/scanner_context.cpp Show resolved Hide resolved
be/src/vec/exec/scan/scanner_context.h Show resolved Hide resolved
be/src/vec/exec/scan/scanner_scheduler.cpp Outdated
MetricUnit::NANOSECONDS);

SimplifiedScanScheduler::SimplifiedScanScheduler(std::string sched_name,
std::shared_ptr<CgroupCpuCtl> cg_cpu_ctl)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: pass by value and use std::move [modernize-pass-by-value]

be/src/vec/exec/scan/scanner_scheduler.cpp:399:

-         : _is_stop(false), _cgroup_cpu_ctl(cg_cpu_ctl), _sched_name(sched_name) {
+         : _is_stop(false), _cgroup_cpu_ctl(cg_cpu_ctl), _sched_name(std::move(sched_name)) {

@zhiqiang-hhhh zhiqiang-hhhh force-pushed the feat-concurrcy-scan branch 2 times, most recently from 9a06917 to 041a17b Compare December 16, 2024 07:36
github-actions[bot]
github-actions bot reviewed Dec 16, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/vec/exec/scan/scanner_context.cpp Show resolved Hide resolved
be/src/vec/exec/scan/scanner_scheduler.cpp Outdated
MetricUnit::NANOSECONDS);

SimplifiedScanScheduler::SimplifiedScanScheduler(std::string sched_name,
std::shared_ptr<CgroupCpuCtl> cg_cpu_ctl)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: pass by value and use std::move [modernize-pass-by-value]

be/src/vec/exec/scan/scanner_scheduler.cpp:397:

- _sched_name(sched_name) {
+ _sched_name(std::move(sched_name)) {

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40800 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 504645872e4216e94f69fc2d4dc418efefbf2a89, data reload: false

------ Round 1 ----------------------------------
q1	17870	8070	7841	7841
q2	2055	185	175	175
q3	10698	1031	1180	1031
q4	10418	781	751	751
q5	7601	2839	2868	2839
q6	241	150	149	149
q7	1028	629	612	612
q8	9355	2015	2010	2010
q9	6518	6147	6228	6147
q10	6961	2329	2350	2329
q11	469	280	256	256
q12	434	239	224	224
q13	17774	2959	2980	2959
q14	242	216	212	212
q15	548	504	491	491
q16	677	592	601	592
q17	1021	651	616	616
q18	7429	6849	6837	6837
q19	1342	1180	1153	1153
q20	480	181	189	181
q21	3684	3088	3106	3088
q22	372	319	307	307
Total cold run time: 107217 ms
Total hot run time: 40800 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7748	8469	7749	7749
q2	326	235	233	233
q3	3072	3100	3030	3030
q4	2163	2039	1959	1959
q5	5912	5927	5902	5902
q6	233	150	152	150
q7	2306	1905	1892	1892
q8	3561	3663	3611	3611
q9	9014	9166	9180	9166
q10	3573	3619	3596	3596
q11	603	505	528	505
q12	815	657	660	657
q13	15392	3098	3160	3098
q14	314	301	304	301
q15	568	505	510	505
q16	721	676	636	636
q17	1921	1746	1698	1698
q18	8782	8123	7731	7731
q19	1843	1681	1697	1681
q20	2128	1926	1973	1926
q21	6035	5842	5828	5828
q22	653	607	586	586
Total cold run time: 77683 ms
Total hot run time: 62440 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 199031 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 504645872e4216e94f69fc2d4dc418efefbf2a89, data reload: false

query1	1316	952	944	944
query2	6241	2495	2567	2495
query3	10974	4728	4774	4728
query4	26483	23958	23885	23885
query5	4369	525	507	507
query6	301	197	197	197
query7	3995	300	307	300
query8	317	247	223	223
query9	8433	2667	2663	2663
query10	475	266	261	261
query11	16024	15419	15255	15255
query12	165	107	121	107
query13	1564	418	396	396
query14	9833	7483	7337	7337
query15	240	181	186	181
query16	7723	473	485	473
query17	1525	656	625	625
query18	2026	345	374	345
query19	208	173	177	173
query20	127	119	124	119
query21	206	107	107	107
query22	4881	4719	4592	4592
query23	34735	33775	33728	33728
query24	9674	2797	2871	2797
query25	619	400	418	400
query26	1331	162	160	160
query27	2457	347	357	347
query28	6292	2448	2461	2448
query29	922	426	436	426
query30	242	157	157	157
query31	1024	871	855	855
query32	97	58	64	58
query33	736	311	309	309
query34	1199	540	538	538
query35	907	783	765	765
query36	1128	963	982	963
query37	148	84	81	81
query38	4344	4232	4377	4232
query39	1546	1456	1460	1456
query40	205	107	104	104
query41	53	51	54	51
query42	123	107	112	107
query43	566	534	531	531
query44	1294	823	824	823
query45	196	177	178	177
query46	1227	780	807	780
query47	2043	1986	2002	1986
query48	486	322	305	305
query49	1024	409	411	409
query50	933	435	434	434
query51	7247	7141	7210	7141
query52	111	94	95	94
query53	275	201	211	201
query54	772	426	439	426
query55	88	82	86	82
query56	272	260	254	254
query57	1340	1202	1223	1202
query58	253	239	235	235
query59	3359	3151	2920	2920
query60	270	254	247	247
query61	115	110	120	110
query62	887	711	698	698
query63	227	234	193	193
query64	4053	685	664	664
query65	3341	3213	3224	3213
query66	806	319	313	313
query67	16326	15618	15573	15573
query68	5740	598	580	580
query69	514	264	270	264
query70	1158	1157	1080	1080
query71	523	249	249	249
query72	6823	3805	4077	3805
query73	907	353	358	353
query74	9729	8973	8857	8857
query75	3776	2684	2704	2684
query76	3801	1339	1454	1339
query77	578	286	274	274
query78	10424	9577	9553	9553
query79	2005	617	621	617
query80	720	445	457	445
query81	517	233	237	233
query82	679	124	121	121
query83	186	156	146	146
query84	236	69	67	67
query85	1354	416	301	301
query86	389	299	290	290
query87	4574	4415	4506	4415
query88	3444	2207	2197	2197
query89	462	308	291	291
query90	2081	194	192	192
query91	147	104	107	104
query92	67	55	61	55
query93	1528	627	612	612
query94	850	301	299	299
query95	371	246	255	246
query96	648	271	263	263
query97	2889	2668	2692	2668
query98	224	198	201	198
query99	1609	1327	1342	1327
Total cold run time: 291672 ms
Total hot run time: 199031 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.68 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 504645872e4216e94f69fc2d4dc418efefbf2a89, data reload: false

query1	0.03	0.03	0.03
query2	0.11	0.04	0.04
query3	0.28	0.06	0.06
query4	1.59	0.08	0.08
query5	0.42	0.41	0.40
query6	1.18	0.66	0.65
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.63	0.55	0.55
query10	0.61	0.63	0.60
query11	0.25	0.13	0.13
query12	0.25	0.13	0.13
query13	0.62	0.61	0.60
query14	2.68	2.71	2.80
query15	1.01	0.88	0.88
query16	0.38	0.38	0.39
query17	1.05	1.04	1.05
query18	0.16	0.16	0.16
query19	2.01	1.84	2.04
query20	0.02	0.04	0.03
query21	15.35	0.68	0.67
query22	3.88	6.90	2.22
query23	16.58	1.96	1.31
query24	5.47	0.31	0.28
query25	0.18	0.09	0.09
query26	0.57	0.18	0.18
query27	0.09	0.09	0.09
query28	11.12	1.09	1.06
query29	12.58	3.46	3.45
query30	0.27	0.05	0.05
query31	2.84	0.43	0.42
query32	3.24	0.51	0.50
query33	3.12	3.06	3.09
query34	16.21	4.38	4.38
query35	4.53	4.41	4.45
query36	0.68	0.50	0.50
query37	0.20	0.17	0.17
query38	0.16	0.15	0.16
query39	0.05	0.04	0.04
query40	0.20	0.14	0.15
query41	0.10	0.06	0.05
query42	0.06	0.05	0.05
query43	0.06	0.05	0.05
Total cold run time: 110.89 s
Total hot run time: 33.68 s

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh zhiqiang-hhhh force-pushed the feat-concurrcy-scan branch 2 times, most recently from e6b5b6c to 01a41c7 Compare December 23, 2024 13:18
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40342 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01a41c777acef54100899009ae1a290bfced0459, data reload: false

------ Round 1 ----------------------------------
q1	17650	8069	7700	7700
q2	2073	178	187	178
q3	10518	1136	1087	1087
q4	10217	804	769	769
q5	7584	2867	2805	2805
q6	238	149	143	143
q7	1039	636	620	620
q8	9428	1980	2009	1980
q9	6538	6017	6054	6017
q10	6965	2321	2306	2306
q11	453	261	257	257
q12	432	225	231	225
q13	17781	3000	2948	2948
q14	254	218	220	218
q15	552	513	501	501
q16	678	606	587	587
q17	989	581	575	575
q18	7541	6760	6714	6714
q19	1347	1165	1158	1158
q20	470	188	191	188
q21	3555	3099	3057	3057
q22	361	328	309	309
Total cold run time: 106663 ms
Total hot run time: 40342 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7709	7672	7634	7634
q2	332	226	232	226
q3	2924	3046	3018	3018
q4	2098	1989	1975	1975
q5	5842	5863	5867	5863
q6	234	147	146	146
q7	2273	1815	1836	1815
q8	3460	3569	3569	3569
q9	8989	9089	9061	9061
q10	3546	3558	3554	3554
q11	606	527	500	500
q12	833	614	643	614
q13	12238	3060	3098	3060
q14	310	278	301	278
q15	541	509	504	504
q16	657	660	652	652
q17	1903	1721	1682	1682
q18	8683	8043	7898	7898
q19	1704	1612	1616	1612
q20	2111	1892	1894	1892
q21	6203	5961	5920	5920
q22	649	616	572	572
Total cold run time: 73845 ms
Total hot run time: 62045 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.78% (10092/26025)
Line Coverage: 29.77% (85126/285920)
Region Coverage: 28.88% (43451/150431)
Branch Coverage: 25.42% (22148/87118)
Coverage Report: http://coverage.selectdb-in.cc/coverage/01a41c777acef54100899009ae1a290bfced0459_01a41c777acef54100899009ae1a290bfced0459/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 197062 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 01a41c777acef54100899009ae1a290bfced0459, data reload: false

query1	1327	1010	978	978
query2	6291	2421	2300	2300
query3	10959	4797	4787	4787
query4	26772	23323	23310	23310
query5	5289	511	485	485
query6	309	194	200	194
query7	4020	300	308	300
query8	303	240	238	238
query9	8499	2738	2724	2724
query10	521	251	261	251
query11	15686	15247	15177	15177
query12	167	101	101	101
query13	1562	423	394	394
query14	10191	6623	6971	6623
query15	239	177	183	177
query16	7867	458	507	458
query17	1526	624	596	596
query18	2043	345	339	339
query19	325	178	197	178
query20	131	125	125	125
query21	211	109	108	108
query22	4894	4821	4744	4744
query23	34810	33663	33631	33631
query24	8652	2720	2658	2658
query25	597	413	399	399
query26	722	155	159	155
query27	2618	347	355	347
query28	5252	2506	2505	2505
query29	832	459	447	447
query30	232	153	154	153
query31	1039	838	828	828
query32	93	61	57	57
query33	704	295	317	295
query34	951	565	543	543
query35	916	750	746	746
query36	1111	943	937	937
query37	134	76	73	73
query38	4288	4180	4206	4180
query39	1512	1473	1483	1473
query40	209	102	100	100
query41	54	45	46	45
query42	112	106	105	105
query43	505	514	483	483
query44	1200	806	820	806
query45	186	168	171	168
query46	1163	755	769	755
query47	2059	1993	1980	1980
query48	472	316	327	316
query49	981	430	423	423
query50	914	419	429	419
query51	7433	7198	7226	7198
query52	112	95	102	95
query53	261	190	198	190
query54	650	419	450	419
query55	83	83	86	83
query56	294	265	241	241
query57	1313	1189	1192	1189
query58	254	231	221	221
query59	3266	3077	3009	3009
query60	297	257	261	257
query61	107	106	104	104
query62	870	685	726	685
query63	227	193	194	193
query64	3763	682	663	663
query65	3234	3197	3205	3197
query66	780	293	297	293
query67	15972	15536	15473	15473
query68	5067	561	622	561
query69	484	259	250	250
query70	1126	1136	1077	1077
query71	491	272	269	269
query72	7071	3959	3903	3903
query73	811	356	364	356
query74	9145	8805	8776	8776
query75	3749	2658	2692	2658
query76	3680	1156	1179	1156
query77	565	301	295	295
query78	10438	9506	9499	9499
query79	1147	626	634	626
query80	756	443	446	443
query81	515	233	221	221
query82	430	122	119	119
query83	201	150	150	150
query84	242	77	69	69
query85	1055	310	306	306
query86	334	293	294	293
query87	4595	4418	4469	4418
query88	2770	2279	2256	2256
query89	421	289	287	287
query90	2020	192	193	192
query91	146	107	107	107
query92	63	51	55	51
query93	1474	594	588	588
query94	828	283	268	268
query95	366	332	255	255
query96	598	278	277	277
query97	2869	2747	2698	2698
query98	218	194	201	194
query99	1536	1320	1326	1320
Total cold run time: 285577 ms
Total hot run time: 197062 ms

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32621 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 80ed26cb37bcf73b5dbf16b729139492be258d10, data reload: false

------ Round 1 ----------------------------------
q1	17808	6593	6407	6407
q2	2051	292	165	165
q3	10564	1269	722	722
q4	10217	936	450	450
q5	7610	2416	2160	2160
q6	201	178	147	147
q7	891	728	618	618
q8	9294	1312	1098	1098
q9	4836	4444	4427	4427
q10	6758	2320	1883	1883
q11	457	283	252	252
q12	348	369	224	224
q13	17786	3637	3033	3033
q14	242	246	223	223
q15	575	517	505	505
q16	620	613	589	589
q17	571	853	340	340
q18	7292	6439	6434	6434
q19	1269	955	575	575
q20	301	314	181	181
q21	2775	2133	1890	1890
q22	358	330	298	298
Total cold run time: 102824 ms
Total hot run time: 32621 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6645	6548	6529	6529
q2	235	333	235	235
q3	2191	2736	2355	2355
q4	1482	1861	1420	1420
q5	4601	4438	4654	4438
q6	232	187	144	144
q7	2047	2027	1906	1906
q8	2692	2872	2710	2710
q9	7322	7390	7494	7390
q10	3063	3243	2821	2821
q11	573	525	500	500
q12	761	797	654	654
q13	3487	3863	3257	3257
q14	293	314	310	310
q15	551	511	493	493
q16	657	686	677	677
q17	1238	1668	1371	1371
q18	8039	7483	7205	7205
q19	801	835	871	835
q20	1882	1977	1869	1869
q21	5581	4932	4725	4725
q22	604	628	575	575
Total cold run time: 54977 ms
Total hot run time: 52419 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.85% (10129/26069)
Line Coverage: 29.91% (85721/286586)
Region Coverage: 29.00% (43726/150762)
Branch Coverage: 25.55% (22322/87362)
Coverage Report: http://coverage.selectdb-in.cc/coverage/80ed26cb37bcf73b5dbf16b729139492be258d10_80ed26cb37bcf73b5dbf16b729139492be258d10/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 190753 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 80ed26cb37bcf73b5dbf16b729139492be258d10, data reload: false

query1	983	379	375	375
query2	6531	2390	2439	2390
query3	6708	209	212	209
query4	27095	23812	23826	23812
query5	4304	673	492	492
query6	301	199	187	187
query7	4638	504	301	301
query8	304	252	242	242
query9	8604	2660	2653	2653
query10	472	296	259	259
query11	15733	15327	15245	15245
query12	168	109	107	107
query13	1675	533	398	398
query14	10625	6943	7220	6943
query15	248	192	190	190
query16	8019	629	416	416
query17	1599	728	559	559
query18	2007	398	299	299
query19	284	179	156	156
query20	124	114	115	114
query21	204	118	101	101
query22	4417	4406	4350	4350
query23	34434	33407	33285	33285
query24	7292	2416	2449	2416
query25	503	450	371	371
query26	1193	273	155	155
query27	2032	493	334	334
query28	3896	2411	2438	2411
query29	711	565	407	407
query30	228	187	153	153
query31	997	915	834	834
query32	74	67	65	65
query33	554	355	305	305
query34	817	843	507	507
query35	798	827	725	725
query36	1001	1019	947	947
query37	128	122	80	80
query38	4156	4327	4136	4136
query39	1530	1437	1456	1437
query40	211	125	106	106
query41	46	47	44	44
query42	122	104	103	103
query43	527	527	519	519
query44	1338	798	799	798
query45	187	169	173	169
query46	912	1055	681	681
query47	1943	1977	1911	1911
query48	398	416	313	313
query49	776	516	431	431
query50	698	752	417	417
query51	7043	6975	6946	6946
query52	104	110	92	92
query53	233	262	187	187
query54	519	487	406	406
query55	80	76	83	76
query56	272	284	259	259
query57	1202	1203	1166	1166
query58	245	231	240	231
query59	2989	3096	2848	2848
query60	285	288	286	286
query61	114	119	108	108
query62	918	800	751	751
query63	238	202	202	202
query64	4222	986	695	695
query65	3276	3187	3218	3187
query66	1049	419	318	318
query67	16054	15797	15417	15417
query68	8854	715	500	500
query69	453	302	263	263
query70	1238	1197	1139	1139
query71	454	298	282	282
query72	5980	3571	3800	3571
query73	802	768	355	355
query74	9257	9607	9116	9116
query75	4110	3154	2712	2712
query76	3478	1185	752	752
query77	768	383	284	284
query78	10093	10307	9434	9434
query79	2694	807	594	594
query80	632	519	435	435
query81	469	274	229	229
query82	678	163	120	120
query83	161	175	159	159
query84	228	90	78	78
query85	791	357	309	309
query86	369	303	297	297
query87	4557	4592	4412	4412
query88	3541	2229	2231	2229
query89	411	328	291	291
query90	1933	189	225	189
query91	141	133	107	107
query92	76	61	59	59
query93	1592	995	571	571
query94	654	396	282	282
query95	358	271	249	249
query96	481	554	266	266
query97	2907	2997	2782	2782
query98	240	212	204	204
query99	1651	1615	1450	1450
Total cold run time: 281439 ms
Total hot run time: 190753 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.91 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 80ed26cb37bcf73b5dbf16b729139492be258d10, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.03
query3	0.24	0.06	0.07
query4	1.62	0.10	0.10
query5	0.43	0.43	0.41
query6	1.16	0.64	0.64
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.61	0.54	0.54
query10	0.60	0.61	0.60
query11	0.14	0.10	0.10
query12	0.14	0.11	0.10
query13	0.59	0.57	0.58
query14	2.83	2.78	2.82
query15	0.91	0.84	0.83
query16	0.38	0.39	0.36
query17	1.07	1.04	1.04
query18	0.21	0.20	0.20
query19	1.92	2.01	1.84
query20	0.02	0.00	0.01
query21	15.37	0.91	0.54
query22	0.75	1.15	0.63
query23	15.00	1.41	0.60
query24	6.68	2.03	0.67
query25	0.47	0.11	0.16
query26	0.71	0.18	0.14
query27	0.05	0.06	0.05
query28	9.24	1.33	0.90
query29	12.56	3.99	3.38
query30	0.25	0.10	0.05
query31	2.82	0.61	0.37
query32	3.23	0.55	0.45
query33	3.06	3.18	3.17
query34	15.84	4.92	4.41
query35	4.38	4.42	4.37
query36	0.70	0.49	0.47
query37	0.09	0.07	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.18	0.15	0.13
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 104.65 s
Total hot run time: 30.91 s

@zhiqiang-hhhh zhiqiang-hhhh marked this pull request as ready for review January 8, 2025 06:30
@zhiqiang-hhhh zhiqiang-hhhh changed the title [WIP](scan) Adaptive scan concurrency [feat](scan) Adaptive scan concurrency Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@zhiqiang-hhhh @doris-robot