Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [Enhancement](cloud) Add concurrency for tablet stats processing #43903 #44763

Merged
merged 1 commit into from
Dec 24, 2024

Conversation

github-actions[bot]
Copy link
Contributor

Cherry-picked from #43903

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Nov 29, 2024
@doris-robot
Copy link

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40340 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35652d1634f7ffced8841cdde1478b380919fb2e, data reload: false

------ Round 1 ----------------------------------
q1	17590	7374	7249	7249
q2	2040	157	178	157
q3	10677	1050	1175	1050
q4	10535	778	704	704
q5	7744	2796	2758	2758
q6	237	147	142	142
q7	948	618	614	614
q8	9595	1906	2003	1906
q9	6721	6382	6366	6366
q10	6978	2228	2310	2228
q11	461	270	261	261
q12	409	219	205	205
q13	17794	2978	2972	2972
q14	254	209	214	209
q15	553	527	527	527
q16	674	605	598	598
q17	952	601	537	537
q18	7186	6577	6604	6577
q19	3526	1054	997	997
q20	478	205	198	198
q21	3920	3107	3163	3107
q22	1062	1004	978	978
Total cold run time: 110334 ms
Total hot run time: 40340 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7412	7274	7321	7274
q2	325	234	225	225
q3	3005	2893	2878	2878
q4	2028	1784	1829	1784
q5	5676	5751	5740	5740
q6	221	140	142	140
q7	2206	1795	1732	1732
q8	3365	3529	3380	3380
q9	8844	9080	9034	9034
q10	3629	3589	3539	3539
q11	603	515	530	515
q12	852	689	676	676
q13	17354	3182	3133	3133
q14	316	276	288	276
q15	573	517	537	517
q16	707	666	672	666
q17	1870	1624	1632	1624
q18	8197	7944	7693	7693
q19	5459	1552	1529	1529
q20	2041	1839	1860	1839
q21	9455	5353	5185	5185
q22	1100	1029	996	996
Total cold run time: 85238 ms
Total hot run time: 60375 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194799 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 35652d1634f7ffced8841cdde1478b380919fb2e, data reload: false

query1	1284	917	878	878
query2	6239	2087	2005	2005
query3	10798	3768	3809	3768
query4	67953	29247	23423	23423
query5	5346	442	420	420
query6	397	195	183	183
query7	5687	304	307	304
query8	301	218	223	218
query9	9232	2668	2655	2655
query10	490	274	267	267
query11	17696	15105	15819	15105
query12	156	111	108	108
query13	1580	418	418	418
query14	10781	6998	6680	6680
query15	212	182	174	174
query16	7607	490	524	490
query17	1099	572	572	572
query18	1949	323	310	310
query19	209	145	148	145
query20	115	107	110	107
query21	211	101	101	101
query22	4534	4395	4368	4368
query23	34141	34293	34062	34062
query24	6059	2953	2930	2930
query25	529	407	412	407
query26	654	162	169	162
query27	1770	302	304	302
query28	4233	2535	2555	2535
query29	699	460	461	460
query30	241	180	156	156
query31	1027	820	822	820
query32	62	51	53	51
query33	400	279	260	260
query34	897	496	487	487
query35	832	720	752	720
query36	1104	957	946	946
query37	112	73	71	71
query38	3972	4017	3939	3939
query39	1528	1466	1471	1466
query40	197	97	95	95
query41	50	48	45	45
query42	113	94	95	94
query43	515	488	489	488
query44	1182	785	792	785
query45	192	167	186	167
query46	1137	744	719	719
query47	1989	1872	1900	1872
query48	451	369	378	369
query49	728	374	375	374
query50	809	406	415	406
query51	7195	7155	7028	7028
query52	103	86	88	86
query53	249	181	182	181
query54	549	457	447	447
query55	74	73	75	73
query56	259	219	224	219
query57	1180	1103	1095	1095
query58	212	197	196	196
query59	3256	3077	3007	3007
query60	275	250	252	250
query61	103	97	104	97
query62	764	663	634	634
query63	206	179	180	179
query64	1470	643	640	640
query65	3235	3133	3196	3133
query66	709	307	297	297
query67	15612	15414	15505	15414
query68	4430	546	542	542
query69	441	247	248	247
query70	1082	1073	1114	1073
query71	404	251	247	247
query72	6489	3875	3887	3875
query73	753	333	336	333
query74	10237	8918	8985	8918
query75	3317	2617	2629	2617
query76	2027	1075	1228	1075
query77	569	260	263	260
query78	10738	9652	9337	9337
query79	8954	596	590	590
query80	2319	413	420	413
query81	547	239	236	236
query82	1468	115	113	113
query83	266	138	137	137
query84	293	73	78	73
query85	1856	300	297	297
query86	477	289	285	285
query87	4526	4240	4259	4240
query88	5579	2408	2406	2406
query89	556	295	288	288
query90	2077	183	181	181
query91	175	141	138	138
query92	66	48	57	48
query93	6799	562	560	560
query94	877	281	270	270
query95	345	254	257	254
query96	666	287	280	280
query97	3393	3106	3162	3106
query98	218	194	194	194
query99	1645	1312	1303	1303
Total cold run time: 338539 ms
Total hot run time: 194799 ms

@Yukang-Lian
Copy link
Collaborator

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40922 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35652d1634f7ffced8841cdde1478b380919fb2e, data reload: false

------ Round 1 ----------------------------------
q1	17904	7489	7353	7353
q2	2062	169	163	163
q3	10630	1126	1189	1126
q4	10251	759	720	720
q5	7721	2883	2764	2764
q6	240	146	144	144
q7	972	614	602	602
q8	9356	1948	2020	1948
q9	6729	6845	6468	6468
q10	7023	2301	2332	2301
q11	462	262	268	262
q12	447	214	217	214
q13	17792	3004	3002	3002
q14	235	206	212	206
q15	565	513	538	513
q16	685	607	597	597
q17	992	573	616	573
q18	7455	6697	6684	6684
q19	2938	1048	1009	1009
q20	470	199	196	196
q21	3875	3173	3116	3116
q22	1090	961	965	961
Total cold run time: 109894 ms
Total hot run time: 40922 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7471	7235	7257	7235
q2	323	227	223	223
q3	2936	2745	2834	2745
q4	2048	1738	1773	1738
q5	5729	5650	5730	5650
q6	224	143	141	141
q7	2124	1704	1744	1704
q8	3379	3621	3512	3512
q9	8957	8585	8860	8585
q10	3569	3535	3561	3535
q11	606	503	495	495
q12	835	628	602	602
q13	17660	3089	3075	3075
q14	289	270	284	270
q15	567	531	506	506
q16	682	667	675	667
q17	1860	1637	1619	1619
q18	8205	7891	7757	7757
q19	4388	1633	1648	1633
q20	2092	1828	1842	1828
q21	5449	5290	5194	5194
q22	1152	1003	997	997
Total cold run time: 80545 ms
Total hot run time: 59711 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191013 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 35652d1634f7ffced8841cdde1478b380919fb2e, data reload: false

query1	967	398	366	366
query2	6504	2122	1985	1985
query3	6698	210	211	210
query4	34267	23452	23449	23449
query5	4236	432	433	432
query6	276	164	166	164
query7	4620	308	296	296
query8	282	215	211	211
query9	9510	2680	2685	2680
query10	485	256	257	256
query11	18085	15122	15253	15122
query12	151	97	102	97
query13	1630	426	398	398
query14	9796	7406	7292	7292
query15	237	174	176	174
query16	7572	456	482	456
query17	1597	590	572	572
query18	2066	309	326	309
query19	219	165	153	153
query20	118	122	115	115
query21	204	99	102	99
query22	4888	4595	4611	4595
query23	35123	34684	34187	34187
query24	10021	2854	2806	2806
query25	575	372	372	372
query26	722	168	163	163
query27	2148	291	289	289
query28	6469	2457	2466	2457
query29	712	428	418	418
query30	321	173	155	155
query31	1073	769	783	769
query32	88	58	52	52
query33	637	294	286	286
query34	889	499	498	498
query35	812	722	715	715
query36	1098	935	936	935
query37	115	77	71	71
query38	3905	3901	3883	3883
query39	1467	1409	1435	1409
query40	209	98	104	98
query41	49	50	48	48
query42	105	97	91	91
query43	521	478	480	478
query44	1185	779	781	779
query45	182	165	166	165
query46	1128	690	705	690
query47	1917	1830	1799	1799
query48	477	368	366	366
query49	1026	379	372	372
query50	834	417	402	402
query51	7423	6980	7152	6980
query52	102	96	86	86
query53	255	175	181	175
query54	886	458	467	458
query55	78	80	79	79
query56	269	238	242	238
query57	1203	1126	1112	1112
query58	231	204	207	204
query59	3257	2934	2764	2764
query60	291	255	254	254
query61	129	130	125	125
query62	814	667	663	663
query63	214	183	180	180
query64	3787	734	707	707
query65	3249	3161	3141	3141
query66	737	307	312	307
query67	15865	15458	15392	15392
query68	4463	573	575	573
query69	457	254	254	254
query70	1164	1121	1143	1121
query71	344	248	248	248
query72	6811	3904	3843	3843
query73	759	335	354	335
query74	9976	8887	8899	8887
query75	3353	2593	2599	2593
query76	2354	976	1017	976
query77	395	262	263	262
query78	10598	9621	9453	9453
query79	9399	586	590	586
query80	2333	409	411	409
query81	582	234	239	234
query82	1694	118	115	115
query83	259	136	140	136
query84	295	79	78	78
query85	2253	294	279	279
query86	488	292	297	292
query87	4513	4253	4237	4237
query88	5793	2455	2451	2451
query89	565	284	289	284
query90	2230	182	181	181
query91	181	143	146	143
query92	61	49	47	47
query93	6862	543	541	541
query94	1117	258	283	258
query95	335	243	240	240
query96	630	286	280	280
query97	3342	3133	3177	3133
query98	251	201	199	199
query99	1714	1300	1296	1296
Total cold run time: 312577 ms
Total hot run time: 191013 ms

@gavinchou gavinchou merged commit 9219689 into branch-3.0 Dec 24, 2024
19 of 21 checks passed
@github-actions github-actions bot deleted the auto-pick-43903-branch-3.0 branch December 24, 2024 10:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants