[refactor][profile] log enhancement and minor change of profile #46564

zhiqiang-hhhh · 2025-01-07T13:45:07Z

What problem does this PR solve?

Add more debug log for profile collection.
When profile need to be remove by TTL, use query finish time instead of query start time, so that we can make our test case more stable.
UT for ProfileManager
RuntimeProfile/Counter/AggCounter is moved to package profile
Modify CI conf of profile.

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-01-07T13:45:13Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zhiqiang-hhhh · 2025-01-07T13:45:16Z

run buildall

zhiqiang-hhhh · 2025-01-07T13:58:40Z

run compile

doris-robot · 2025-01-07T14:05:43Z

TPC-H: Total hot run time: 32839 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit af7a871603f81985dc6063e480e8e3cd26737d3f, data reload: false

------ Round 1 ----------------------------------
q1	17633	6178	6081	6081
q2	2057	291	161	161
q3	10430	1252	728	728
q4	10222	857	427	427
q5	7498	2143	2002	2002
q6	196	177	146	146
q7	898	751	603	603
q8	9233	1318	1184	1184
q9	5235	4939	4982	4939
q10	6747	2308	1860	1860
q11	466	276	261	261
q12	356	351	227	227
q13	17777	3728	2983	2983
q14	242	231	208	208
q15	567	516	500	500
q16	641	638	591	591
q17	599	854	344	344
q18	7201	6506	6469	6469
q19	1231	962	571	571
q20	308	318	194	194
q21	2910	2173	2052	2052
q22	361	332	308	308
Total cold run time: 102808 ms
Total hot run time: 32839 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6194	6244	6226	6226
q2	233	321	234	234
q3	2214	2652	2354	2354
q4	1373	1813	1387	1387
q5	4362	4789	4768	4768
q6	188	184	147	147
q7	2125	1982	1886	1886
q8	2608	2843	2679	2679
q9	7342	7147	7191	7147
q10	3032	3320	2826	2826
q11	602	534	497	497
q12	665	741	595	595
q13	3488	3902	3192	3192
q14	279	315	281	281
q15	562	521	502	502
q16	659	685	640	640
q17	1209	1730	1259	1259
q18	7739	7414	7200	7200
q19	828	1104	1059	1059
q20	1984	1961	1798	1798
q21	5479	5111	4949	4949
q22	597	590	578	578
Total cold run time: 53762 ms
Total hot run time: 52204 ms

doris-robot · 2025-01-07T14:17:48Z

TPC-DS: Total hot run time: 192277 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit af7a871603f81985dc6063e480e8e3cd26737d3f, data reload: false

query1	955	394	399	394
query2	6517	2472	2467	2467
query3	6716	215	209	209
query4	34419	24444	23949	23949
query5	4344	625	462	462
query6	275	201	195	195
query7	4622	511	309	309
query8	311	252	238	238
query9	9573	2771	2766	2766
query10	470	316	258	258
query11	18469	15874	15492	15492
query12	159	114	107	107
query13	1667	557	426	426
query14	10198	8190	6847	6847
query15	232	199	187	187
query16	7716	576	450	450
query17	1573	725	554	554
query18	1259	370	321	321
query19	216	182	160	160
query20	123	114	112	112
query21	209	129	107	107
query22	4487	4498	4303	4303
query23	35028	33540	33412	33412
query24	6462	2369	2372	2369
query25	492	452	380	380
query26	1205	276	157	157
query27	1990	482	343	343
query28	4982	2473	2445	2445
query29	615	537	413	413
query30	231	189	162	162
query31	967	886	823	823
query32	76	59	60	59
query33	501	362	296	296
query34	764	859	527	527
query35	836	827	724	724
query36	1024	1047	970	970
query37	122	101	75	75
query38	4357	4239	4042	4042
query39	1499	1482	1474	1474
query40	210	115	102	102
query41	45	48	44	44
query42	123	106	100	100
query43	512	524	482	482
query44	1371	801	811	801
query45	183	173	178	173
query46	873	1062	667	667
query47	1946	1955	1850	1850
query48	390	418	329	329
query49	773	489	392	392
query50	629	651	402	402
query51	7210	7129	6875	6875
query52	107	102	94	94
query53	230	256	189	189
query54	471	481	410	410
query55	81	74	79	74
query56	258	262	247	247
query57	1248	1210	1125	1125
query58	240	241	233	233
query59	3028	3084	2880	2880
query60	269	266	247	247
query61	110	111	108	108
query62	853	792	716	716
query63	240	199	203	199
query64	4128	993	640	640
query65	3460	3298	3281	3281
query66	1086	443	335	335
query67	15978	15816	15713	15713
query68	4966	743	540	540
query69	487	313	287	287
query70	1265	1131	1132	1131
query71	401	296	260	260
query72	6320	4072	3959	3959
query73	652	780	372	372
query74	10579	9035	9066	9035
query75	3246	3156	2643	2643
query76	3268	1226	796	796
query77	462	378	286	286
query78	10628	10520	9809	9809
query79	3519	851	597	597
query80	714	527	438	438
query81	507	273	245	245
query82	583	158	123	123
query83	174	170	153	153
query84	247	95	70	70
query85	828	352	308	308
query86	389	307	291	291
query87	4502	4484	4435	4435
query88	4984	2209	2214	2209
query89	419	335	312	312
query90	1835	193	202	193
query91	137	133	105	105
query92	73	57	57	57
query93	2331	869	550	550
query94	706	385	282	282
query95	333	274	253	253
query96	489	621	293	293
query97	3004	3073	2894	2894
query98	240	231	198	198
query99	1494	1563	1461	1461
Total cold run time: 291495 ms
Total hot run time: 192277 ms

doris-robot · 2025-01-07T14:23:07Z

ClickBench: Total hot run time: 31.19 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit af7a871603f81985dc6063e480e8e3cd26737d3f, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.04
query3	0.24	0.07	0.07
query4	1.61	0.11	0.10
query5	0.43	0.40	0.40
query6	1.14	0.66	0.65
query7	0.03	0.02	0.01
query8	0.04	0.04	0.03
query9	0.56	0.49	0.51
query10	0.56	0.58	0.57
query11	0.15	0.10	0.10
query12	0.13	0.11	0.11
query13	0.61	0.60	0.60
query14	2.70	2.86	2.84
query15	0.89	0.82	0.83
query16	0.36	0.41	0.38
query17	1.06	1.04	1.05
query18	0.24	0.21	0.22
query19	1.83	1.89	2.00
query20	0.01	0.01	0.01
query21	15.36	0.93	0.56
query22	0.75	0.86	0.60
query23	15.31	1.42	0.58
query24	2.49	0.99	0.71
query25	0.23	0.08	0.12
query26	0.29	0.18	0.13
query27	0.05	0.07	0.05
query28	13.59	1.58	1.06
query29	12.59	3.91	3.30
query30	0.26	0.09	0.06
query31	2.82	0.59	0.38
query32	3.23	0.54	0.46
query33	3.04	3.14	3.22
query34	16.77	5.06	4.46
query35	4.52	4.45	4.47
query36	0.65	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.13 s
Total hot run time: 31.19 s

doris-robot · 2025-01-07T14:27:03Z

TeamCity be ut coverage result:
Function Coverage: 38.86% (10128/26063)
Line Coverage: 29.91% (85708/286532)
Region Coverage: 29.01% (43725/150742)
Branch Coverage: 25.55% (22319/87362)
Coverage Report: http://coverage.selectdb-in.cc/coverage/af7a871603f81985dc6063e480e8e3cd26737d3f_af7a871603f81985dc6063e480e8e3cd26737d3f/report/index.html

zhiqiang-hhhh · 2025-01-07T15:10:21Z

run p0 10

shuke987 · 2025-01-08T03:58:29Z

run p0

shuke987 · 2025-01-08T07:14:33Z

run p0

shuke987 · 2025-01-08T08:14:46Z

run p0

zhiqiang-hhhh · 2025-01-09T13:08:59Z

run buildall

doris-robot · 2025-01-09T14:05:07Z

TeamCity be ut coverage result:
Function Coverage: 39.34% (10252/26058)
Line Coverage: 30.50% (87364/286399)
Region Coverage: 29.55% (44534/150688)
Branch Coverage: 26.11% (22800/87322)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b6abfb7813d5267d8a895dea6c3db9284e490d05_b6abfb7813d5267d8a895dea6c3db9284e490d05/report/index.html

zhiqiang-hhhh · 2025-01-09T15:41:57Z

run buildall

doris-robot · 2025-01-09T16:06:07Z

TPC-H: Total hot run time: 32316 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f6500eb047b7adbedafad49f2ca1d59af7c2c2e0, data reload: false

------ Round 1 ----------------------------------
q1	17595	6093	5978	5978
q2	2040	303	168	168
q3	10418	1233	704	704
q4	10208	860	428	428
q5	7530	2124	1932	1932
q6	201	177	150	150
q7	872	740	591	591
q8	9237	1345	1130	1130
q9	5118	4873	4847	4847
q10	6739	2281	1849	1849
q11	470	283	255	255
q12	344	351	220	220
q13	17784	3673	3081	3081
q14	222	223	211	211
q15	559	505	486	486
q16	628	617	578	578
q17	566	849	330	330
q18	7155	6498	6350	6350
q19	1228	930	542	542
q20	311	316	190	190
q21	2770	2195	1987	1987
q22	373	337	309	309
Total cold run time: 102368 ms
Total hot run time: 32316 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6193	6198	6158	6158
q2	232	325	241	241
q3	2279	2581	2331	2331
q4	1395	1788	1404	1404
q5	4292	4721	4574	4574
q6	185	176	141	141
q7	2088	1947	1850	1850
q8	2592	2796	2651	2651
q9	7228	7127	7229	7127
q10	2977	3310	2834	2834
q11	581	497	505	497
q12	676	762	597	597
q13	3423	3729	3121	3121
q14	274	289	272	272
q15	570	499	506	499
q16	637	672	641	641
q17	1186	1724	1230	1230
q18	7675	7649	7051	7051
q19	715	1039	1021	1021
q20	1932	1927	1868	1868
q21	5367	4936	4716	4716
q22	587	626	588	588
Total cold run time: 53084 ms
Total hot run time: 51412 ms

doris-robot · 2025-01-09T16:18:00Z

TPC-DS: Total hot run time: 187155 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f6500eb047b7adbedafad49f2ca1d59af7c2c2e0, data reload: false

query1	981	384	381	381
query2	6520	2321	2326	2321
query3	6722	211	217	211
query4	33565	23286	23241	23241
query5	4319	600	457	457
query6	284	220	182	182
query7	4633	483	299	299
query8	294	241	233	233
query9	9634	2703	2685	2685
query10	484	314	242	242
query11	17855	15164	15106	15106
query12	148	108	107	107
query13	1650	505	389	389
query14	10306	6845	6341	6341
query15	255	210	190	190
query16	8177	604	462	462
query17	1597	735	583	583
query18	2099	425	291	291
query19	222	169	149	149
query20	118	117	120	117
query21	212	157	103	103
query22	4177	4421	4091	4091
query23	33985	32825	32983	32825
query24	6435	2183	2268	2183
query25	487	447	384	384
query26	1207	272	150	150
query27	2133	449	348	348
query28	5379	2440	2448	2440
query29	697	529	406	406
query30	221	185	157	157
query31	983	891	787	787
query32	86	60	60	60
query33	497	359	278	278
query34	740	835	492	492
query35	772	781	719	719
query36	1025	991	957	957
query37	129	94	75	75
query38	3985	4101	4015	4015
query39	1484	1409	1407	1407
query40	209	111	97	97
query41	50	49	51	49
query42	121	104	100	100
query43	508	514	494	494
query44	1313	835	804	804
query45	182	167	161	161
query46	857	1040	643	643
query47	1871	1876	1814	1814
query48	381	393	313	313
query49	784	478	387	387
query50	637	631	371	371
query51	6880	6923	6906	6906
query52	99	97	88	88
query53	219	244	182	182
query54	475	487	409	409
query55	81	82	80	80
query56	250	266	235	235
query57	1177	1161	1091	1091
query58	234	222	241	222
query59	2958	3072	2982	2982
query60	292	284	240	240
query61	139	112	111	111
query62	882	744	708	708
query63	215	193	196	193
query64	4428	1017	639	639
query65	3273	3131	3144	3131
query66	1044	420	320	320
query67	15901	15810	15456	15456
query68	7786	697	517	517
query69	468	282	255	255
query70	1233	1107	1145	1107
query71	425	288	247	247
query72	6143	3783	3813	3783
query73	656	738	354	354
query74	10282	8973	8702	8702
query75	4089	3167	2636	2636
query76	3850	1159	766	766
query77	771	365	281	281
query78	9871	9910	9322	9322
query79	3264	804	590	590
query80	631	518	450	450
query81	462	275	221	221
query82	592	155	123	123
query83	213	170	148	148
query84	288	92	72	72
query85	774	338	364	338
query86	351	307	306	306
query87	4730	4296	4487	4296
query88	4284	2181	2114	2114
query89	401	318	293	293
query90	1916	189	186	186
query91	145	138	108	108
query92	63	56	53	53
query93	1223	843	533	533
query94	656	401	295	295
query95	331	261	260	260
query96	480	612	277	277
query97	2868	2901	2842	2842
query98	213	194	194	194
query99	1605	1517	1396	1396
Total cold run time: 291021 ms
Total hot run time: 187155 ms

doris-robot · 2025-01-09T16:23:19Z

ClickBench: Total hot run time: 30.98 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f6500eb047b7adbedafad49f2ca1d59af7c2c2e0, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.03
query3	0.24	0.07	0.07
query4	1.61	0.11	0.10
query5	0.43	0.42	0.41
query6	1.16	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.59	0.49	0.52
query10	0.55	0.57	0.55
query11	0.14	0.10	0.10
query12	0.15	0.12	0.11
query13	0.61	0.60	0.60
query14	2.72	2.71	2.74
query15	0.88	0.83	0.83
query16	0.40	0.37	0.38
query17	1.05	0.99	1.00
query18	0.23	0.21	0.20
query19	1.87	1.79	1.96
query20	0.01	0.01	0.01
query21	15.39	0.90	0.58
query22	0.76	0.84	0.64
query23	15.26	1.37	0.54
query24	2.77	0.80	1.52
query25	0.22	0.22	0.15
query26	0.20	0.15	0.13
query27	0.06	0.06	0.04
query28	13.55	1.54	1.05
query29	12.55	3.98	3.26
query30	0.25	0.09	0.06
query31	2.83	0.62	0.39
query32	3.22	0.54	0.47
query33	3.11	3.24	3.08
query34	16.75	5.09	4.45
query35	4.48	4.46	4.50
query36	0.66	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 105.37 s
Total hot run time: 30.98 s

doris-robot · 2025-01-09T16:32:28Z

TeamCity be ut coverage result:
Function Coverage: 39.34% (10252/26058)
Line Coverage: 30.52% (87406/286402)
Region Coverage: 29.56% (44541/150688)
Branch Coverage: 26.12% (22807/87322)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f6500eb047b7adbedafad49f2ca1d59af7c2c2e0_f6500eb047b7adbedafad49f2ca1d59af7c2c2e0/report/index.html

doris-robot · 2025-01-10T04:01:36Z

TeamCity be ut coverage result:
Function Coverage: 39.34% (10252/26058)
Line Coverage: 30.51% (87372/286402)
Region Coverage: 29.56% (44546/150688)
Branch Coverage: 26.11% (22804/87322)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6fce396c9dd46c1b82ecc3948370bf965ae3ecec_6fce396c9dd46c1b82ecc3948370bf965ae3ecec/report/index.html

zhiqiang-hhhh · 2025-01-10T04:27:36Z

run buildall

doris-robot · 2025-01-10T05:08:49Z

TeamCity be ut coverage result:
Function Coverage: 39.34% (10250/26058)
Line Coverage: 30.50% (87357/286402)
Region Coverage: 29.55% (44527/150688)
Branch Coverage: 26.11% (22799/87322)
Coverage Report: http://coverage.selectdb-in.cc/coverage/40d87f2b37ca0a8c0b45d879fcd97907c0ed98fa_40d87f2b37ca0a8c0b45d879fcd97907c0ed98fa/report/index.html

doris-robot · 2025-01-10T05:13:24Z

TPC-H: Total hot run time: 32811 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 40d87f2b37ca0a8c0b45d879fcd97907c0ed98fa, data reload: false

------ Round 1 ----------------------------------
q1	17598	6175	6058	6058
q2	2048	304	184	184
q3	10569	1256	753	753
q4	10233	875	446	446
q5	7825	2222	1996	1996
q6	211	181	152	152
q7	884	767	619	619
q8	9231	1386	1206	1206
q9	5233	4959	4985	4959
q10	7090	2329	1864	1864
q11	487	283	270	270
q12	355	363	234	234
q13	17805	3682	3111	3111
q14	251	238	216	216
q15	550	501	495	495
q16	645	634	568	568
q17	598	849	345	345
q18	6889	6344	6268	6268
q19	2268	955	576	576
q20	308	320	196	196
q21	3003	2265	1987	1987
q22	357	330	308	308
Total cold run time: 104438 ms
Total hot run time: 32811 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6388	6244	6409	6244
q2	247	336	242	242
q3	2246	2691	2326	2326
q4	1410	1760	1367	1367
q5	4336	4774	4856	4774
q6	188	179	142	142
q7	2064	1930	1803	1803
q8	2606	2806	2672	2672
q9	7285	7206	7209	7206
q10	3103	3299	2764	2764
q11	583	530	493	493
q12	723	766	620	620
q13	3447	3874	3188	3188
q14	291	319	286	286
q15	564	501	509	501
q16	640	721	642	642
q17	1233	1729	1292	1292
q18	7754	7460	7397	7397
q19	844	1166	1041	1041
q20	1972	2079	1842	1842
q21	5639	5125	5039	5039
q22	632	638	568	568
Total cold run time: 54195 ms
Total hot run time: 52449 ms

doris-robot · 2025-01-10T05:25:39Z

TPC-DS: Total hot run time: 194411 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 40d87f2b37ca0a8c0b45d879fcd97907c0ed98fa, data reload: false

query1	1298	956	939	939
query2	6387	2385	2303	2303
query3	11062	4867	4838	4838
query4	32391	23750	23148	23148
query5	3841	600	444	444
query6	292	200	174	174
query7	3979	490	304	304
query8	288	237	221	221
query9	9474	2729	2740	2729
query10	463	314	242	242
query11	17946	15184	15131	15131
query12	156	112	103	103
query13	1562	521	414	414
query14	9007	6686	6676	6676
query15	254	221	199	199
query16	8196	615	433	433
query17	1614	782	595	595
query18	2138	421	331	331
query19	198	184	164	164
query20	118	126	116	116
query21	210	127	110	110
query22	4588	4442	4298	4298
query23	34324	33338	33181	33181
query24	6592	2300	2279	2279
query25	493	498	400	400
query26	1215	274	154	154
query27	2591	469	331	331
query28	5829	2483	2494	2483
query29	751	550	424	424
query30	218	188	161	161
query31	940	849	797	797
query32	94	57	59	57
query33	483	371	301	301
query34	776	849	537	537
query35	809	819	727	727
query36	1025	1053	967	967
query37	130	97	74	74
query38	4059	4148	4200	4148
query39	1469	1477	1489	1477
query40	208	116	100	100
query41	52	49	50	49
query42	126	106	105	105
query43	516	527	508	508
query44	1357	841	829	829
query45	184	174	165	165
query46	895	1057	658	658
query47	1922	1904	1820	1820
query48	391	404	315	315
query49	753	502	387	387
query50	662	675	415	415
query51	7096	6979	6865	6865
query52	102	100	93	93
query53	235	251	179	179
query54	473	519	436	436
query55	81	87	83	83
query56	270	252	238	238
query57	1235	1169	1182	1169
query58	249	233	226	226
query59	3123	3219	3056	3056
query60	284	273	261	261
query61	115	108	118	108
query62	850	784	748	748
query63	229	196	187	187
query64	4326	1021	649	649
query65	3351	3229	3264	3229
query66	1114	413	311	311
query67	16475	15625	15430	15430
query68	9435	695	526	526
query69	465	304	247	247
query70	1197	1094	1104	1094
query71	413	282	252	252
query72	6452	3900	3996	3900
query73	650	749	368	368
query74	10206	8850	8779	8779
query75	4358	3158	2645	2645
query76	4026	1193	771	771
query77	766	350	290	290
query78	9934	9847	9416	9416
query79	4225	795	584	584
query80	687	522	432	432
query81	468	288	239	239
query82	531	153	121	121
query83	195	173	151	151
query84	287	90	75	75
query85	813	360	302	302
query86	421	324	291	291
query87	4517	4265	4465	4265
query88	3502	2232	2186	2186
query89	452	325	296	296
query90	1963	193	189	189
query91	132	136	108	108
query92	64	53	50	50
query93	2581	868	531	531
query94	665	401	277	277
query95	341	263	247	247
query96	485	638	274	274
query97	2974	2938	2802	2802
query98	226	205	202	202
query99	1614	1509	1472	1472
Total cold run time: 298979 ms
Total hot run time: 194411 ms

doris-robot · 2025-01-10T05:30:59Z

ClickBench: Total hot run time: 31.22 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 40d87f2b37ca0a8c0b45d879fcd97907c0ed98fa, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.61	0.10	0.11
query5	0.43	0.43	0.40
query6	1.17	0.66	0.66
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.52	0.50
query10	0.57	0.56	0.56
query11	0.14	0.11	0.11
query12	0.14	0.10	0.12
query13	0.60	0.59	0.61
query14	2.82	2.77	2.78
query15	0.89	0.83	0.83
query16	0.37	0.38	0.38
query17	1.04	1.01	1.08
query18	0.23	0.21	0.20
query19	1.90	1.89	2.03
query20	0.01	0.01	0.02
query21	15.37	1.00	0.58
query22	0.75	0.80	0.60
query23	15.31	1.38	0.50
query24	3.36	0.72	1.22
query25	0.21	0.13	0.17
query26	0.30	0.15	0.13
query27	0.06	0.06	0.05
query28	13.17	1.51	1.04
query29	12.57	3.92	3.23
query30	0.26	0.10	0.07
query31	2.82	0.56	0.38
query32	3.23	0.55	0.45
query33	3.14	3.26	3.27
query34	16.84	5.12	4.51
query35	4.58	4.48	4.49
query36	0.64	0.49	0.49
query37	0.10	0.07	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.14	0.12
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 106.03 s
Total hot run time: 31.22 s

zhiqiang-hhhh · 2025-01-10T09:35:46Z

run buildall

doris-robot · 2025-01-10T10:15:45Z

TPC-H: Total hot run time: 32556 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 40d87f2b37ca0a8c0b45d879fcd97907c0ed98fa, data reload: false

------ Round 1 ----------------------------------
q1	17584	6190	6001	6001
q2	2060	302	184	184
q3	10395	1197	746	746
q4	10199	853	433	433
q5	7514	2196	1957	1957
q6	202	178	154	154
q7	905	770	617	617
q8	9262	1365	1153	1153
q9	5119	4838	4943	4838
q10	6808	2320	1882	1882
q11	507	280	272	272
q12	343	370	223	223
q13	18243	3709	3067	3067
q14	238	236	208	208
q15	560	507	508	507
q16	606	620	581	581
q17	587	839	334	334
q18	6833	6373	6460	6373
q19	1778	948	565	565
q20	323	320	190	190
q21	3095	2176	1971	1971
q22	359	338	300	300
Total cold run time: 103520 ms
Total hot run time: 32556 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6236	6226	6254	6226
q2	249	326	237	237
q3	2259	2647	2336	2336
q4	1368	1838	1368	1368
q5	4355	4726	4805	4726
q6	189	181	148	148
q7	2025	1944	1798	1798
q8	2598	2868	2759	2759
q9	7271	7228	7220	7220
q10	3067	3356	2703	2703
q11	574	522	502	502
q12	684	810	622	622
q13	3528	3878	3175	3175
q14	302	303	274	274
q15	583	531	495	495
q16	683	723	640	640
q17	1248	1746	1285	1285
q18	7663	7548	7323	7323
q19	828	1183	1079	1079
q20	1979	2036	1973	1973
q21	5779	5145	5009	5009
q22	622	627	581	581
Total cold run time: 54090 ms
Total hot run time: 52479 ms

doris-robot · 2025-01-10T10:22:42Z

TeamCity be ut coverage result:
Function Coverage: 39.36% (10256/26059)
Line Coverage: 30.52% (87410/286406)
Region Coverage: 29.56% (44545/150688)
Branch Coverage: 26.11% (22803/87322)
Coverage Report: http://coverage.selectdb-in.cc/coverage/40d87f2b37ca0a8c0b45d879fcd97907c0ed98fa_40d87f2b37ca0a8c0b45d879fcd97907c0ed98fa/report/index.html

doris-robot · 2025-01-10T10:28:02Z

TPC-DS: Total hot run time: 195478 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 40d87f2b37ca0a8c0b45d879fcd97907c0ed98fa, data reload: false

query1	1299	969	923	923
query2	6061	2348	2345	2345
query3	11138	4763	4905	4763
query4	32971	23873	23218	23218
query5	4867	608	461	461
query6	294	192	175	175
query7	3976	493	308	308
query8	289	237	222	222
query9	9546	2765	2745	2745
query10	473	301	247	247
query11	17732	15237	15054	15054
query12	153	107	100	100
query13	1564	495	376	376
query14	11179	7659	7493	7493
query15	235	219	194	194
query16	8208	637	480	480
query17	1479	756	582	582
query18	2087	406	300	300
query19	213	195	161	161
query20	120	109	111	109
query21	203	128	102	102
query22	4357	4565	4385	4385
query23	34267	33452	33803	33452
query24	6407	2489	2354	2354
query25	527	487	406	406
query26	808	284	162	162
query27	2137	484	326	326
query28	5507	2487	2515	2487
query29	654	579	446	446
query30	219	195	156	156
query31	1033	918	853	853
query32	93	60	55	55
query33	508	356	299	299
query34	763	865	508	508
query35	797	811	752	752
query36	1014	1053	955	955
query37	126	100	85	85
query38	4114	4127	4302	4127
query39	1520	1427	1414	1414
query40	213	115	104	104
query41	62	57	56	56
query42	131	104	101	101
query43	523	523	508	508
query44	1447	855	844	844
query45	188	182	173	173
query46	891	1057	650	650
query47	1956	1937	1822	1822
query48	388	405	313	313
query49	714	485	394	394
query50	646	683	399	399
query51	7053	7123	6909	6909
query52	104	101	100	100
query53	231	254	189	189
query54	478	510	433	433
query55	90	84	77	77
query56	267	270	246	246
query57	1219	1222	1151	1151
query58	258	229	245	229
query59	3294	3227	3101	3101
query60	308	293	263	263
query61	118	149	112	112
query62	875	832	747	747
query63	240	200	194	194
query64	3613	987	701	701
query65	3307	3268	3350	3268
query66	831	416	312	312
query67	16076	15638	15351	15351
query68	8236	707	524	524
query69	477	292	254	254
query70	1178	1175	1137	1137
query71	405	284	247	247
query72	6190	3858	3853	3853
query73	650	756	366	366
query74	10350	8915	8647	8647
query75	3416	3151	2683	2683
query76	3425	1189	754	754
query77	566	360	274	274
query78	10055	9879	9440	9440
query79	3864	802	585	585
query80	744	501	441	441
query81	511	270	243	243
query82	677	150	123	123
query83	172	165	148	148
query84	250	104	76	76
query85	831	409	295	295
query86	385	304	294	294
query87	4568	4619	4258	4258
query88	4942	2175	2161	2161
query89	421	322	279	279
query90	1817	190	186	186
query91	152	126	109	109
query92	72	53	51	51
query93	2360	861	518	518
query94	675	384	287	287
query95	332	259	250	250
query96	503	602	283	283
query97	2895	2948	2764	2764
query98	223	201	206	201
query99	1592	1531	1420	1420
Total cold run time: 297393 ms
Total hot run time: 195478 ms

doris-robot · 2025-01-10T10:33:24Z

ClickBench: Total hot run time: 31.21 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 40d87f2b37ca0a8c0b45d879fcd97907c0ed98fa, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.61	0.11	0.11
query5	0.43	0.43	0.39
query6	1.15	0.65	0.64
query7	0.03	0.02	0.01
query8	0.04	0.03	0.04
query9	0.59	0.50	0.50
query10	0.56	0.56	0.55
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.60	0.61	0.59
query14	2.70	2.87	2.82
query15	0.90	0.83	0.82
query16	0.37	0.39	0.39
query17	1.05	1.00	1.03
query18	0.22	0.21	0.20
query19	1.88	1.88	1.99
query20	0.01	0.01	0.01
query21	15.73	0.93	0.58
query22	0.99	0.97	0.73
query23	15.17	1.40	0.59
query24	3.89	1.06	0.65
query25	0.21	0.19	0.12
query26	0.21	0.15	0.14
query27	0.08	0.07	0.07
query28	13.84	1.54	1.05
query29	12.59	3.91	3.23
query30	0.26	0.09	0.07
query31	2.82	0.59	0.38
query32	3.23	0.55	0.45
query33	3.13	3.02	3.03
query34	16.83	5.22	4.58
query35	4.53	4.49	4.53
query36	0.79	0.48	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.53 s
Total hot run time: 31.21 s

yiguolei · 2025-01-10T11:26:45Z

fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java

+        } catch (Exception e) {
+            LOG.warn("Failed to submit profile write task, query {} be {}",
+                                DebugUtil.printId(profile.query_id), address.toString());
+            return new Status(TStatusCode.INTERNAL_ERROR, "Failed to submit profile write task");


如果这里返回失败，会不会BE 收到这个错误，导致查询失败？

如果这里返回失败，会不会BE 收到这个错误，导致查询失败？

不会，referenct

doris/fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java

Line 238 in 43f6197

// actually be does not care the process status of profile on fe.

yiguolei · 2025-01-10T11:36:26Z

be/src/runtime/runtime_query_statistics_mgr.cpp

@@ -65,6 +65,8 @@ static Status _do_report_exec_stats_rpc(const TNetworkAddress& coor_addr,
        return Status::RpcError("Client rpc client failed");
    }

+    VLOG_DEBUG << fmt::format("Sending profile");


not need format

zhiqiang-hhhh · 2025-01-10T17:30:20Z

run buildall

doris-robot · 2025-01-10T18:10:03Z

TPC-H: Total hot run time: 32500 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6fe1ca2ba7745ef4f40135925cae1e349038cb9f, data reload: false

------ Round 1 ----------------------------------
q1	17583	6091	6041	6041
q2	2051	308	180	180
q3	10537	1201	740	740
q4	10221	858	424	424
q5	7898	2180	1956	1956
q6	200	179	144	144
q7	894	756	597	597
q8	9242	1363	1190	1190
q9	5280	4878	4913	4878
q10	6792	2306	1864	1864
q11	489	282	273	273
q12	341	364	221	221
q13	17786	3625	3027	3027
q14	225	230	214	214
q15	562	511	517	511
q16	627	613	599	599
q17	553	852	322	322
q18	6792	6336	6458	6336
q19	2099	957	552	552
q20	304	310	196	196
q21	2876	2199	1929	1929
q22	362	332	306	306
Total cold run time: 103714 ms
Total hot run time: 32500 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6321	6207	6192	6192
q2	238	326	235	235
q3	2251	2650	2276	2276
q4	1402	1805	1354	1354
q5	4271	4721	4807	4721
q6	177	175	145	145
q7	2025	1997	1810	1810
q8	2610	2816	2676	2676
q9	7238	7216	7249	7216
q10	3039	3356	2774	2774
q11	580	527	491	491
q12	648	780	628	628
q13	3432	3932	3236	3236
q14	310	291	296	291
q15	578	513	500	500
q16	637	720	669	669
q17	1215	1737	1291	1291
q18	7703	7458	7306	7306
q19	831	1165	1075	1075
q20	2035	2039	1918	1918
q21	5701	5196	4820	4820
q22	612	628	602	602
Total cold run time: 53854 ms
Total hot run time: 52226 ms

doris-robot · 2025-01-10T18:22:21Z

TPC-DS: Total hot run time: 195456 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6fe1ca2ba7745ef4f40135925cae1e349038cb9f, data reload: false

query1	1320	921	944	921
query2	6413	2374	2269	2269
query3	10990	4861	4777	4777
query4	33020	23496	23802	23496
query5	4791	637	453	453
query6	300	200	206	200
query7	3987	499	320	320
query8	312	256	237	237
query9	9465	2664	2678	2664
query10	485	303	269	269
query11	17632	15342	15033	15033
query12	166	106	107	106
query13	1632	515	400	400
query14	11333	7484	7257	7257
query15	254	208	186	186
query16	8732	640	471	471
query17	1538	756	576	576
query18	2149	416	311	311
query19	202	191	153	153
query20	122	118	112	112
query21	206	122	101	101
query22	4479	4422	4356	4356
query23	34071	33115	33369	33115
query24	6300	2391	2414	2391
query25	521	466	393	393
query26	765	284	159	159
query27	2102	474	329	329
query28	5222	2492	2434	2434
query29	682	584	427	427
query30	209	185	163	163
query31	959	875	807	807
query32	93	61	56	56
query33	503	363	298	298
query34	770	871	506	506
query35	810	815	749	749
query36	989	1053	956	956
query37	127	101	82	82
query38	4242	4200	4294	4200
query39	1511	1486	1489	1486
query40	200	123	101	101
query41	53	51	48	48
query42	123	105	100	100
query43	530	536	487	487
query44	1373	823	801	801
query45	189	177	173	173
query46	891	1067	647	647
query47	1847	1900	1842	1842
query48	395	425	336	336
query49	753	491	417	417
query50	679	676	400	400
query51	7083	6992	7033	6992
query52	104	104	92	92
query53	232	267	187	187
query54	509	504	412	412
query55	85	88	82	82
query56	271	255	248	248
query57	1203	1236	1171	1171
query58	242	231	227	227
query59	3220	3384	3117	3117
query60	287	281	261	261
query61	118	115	116	115
query62	845	855	735	735
query63	229	188	188	188
query64	3558	1057	664	664
query65	3310	3339	3203	3203
query66	878	416	310	310
query67	16268	15648	15436	15436
query68	8587	720	515	515
query69	485	284	250	250
query70	1152	1098	1148	1098
query71	439	281	253	253
query72	6312	3810	3841	3810
query73	660	773	356	356
query74	10301	8836	9202	8836
query75	4155	3151	2670	2670
query76	3698	1201	781	781
query77	769	366	274	274
query78	10125	9949	9475	9475
query79	3743	816	575	575
query80	731	636	442	442
query81	511	292	236	236
query82	634	162	123	123
query83	169	165	149	149
query84	249	98	77	77
query85	803	367	299	299
query86	383	319	300	300
query87	4533	4433	4533	4433
query88	4744	2122	2114	2114
query89	457	339	291	291
query90	1804	192	186	186
query91	134	138	108	108
query92	68	61	54	54
query93	2169	868	527	527
query94	644	408	280	280
query95	342	272	256	256
query96	492	618	286	286
query97	2776	2990	2797	2797
query98	222	195	194	194
query99	1586	1497	1400	1400
Total cold run time: 298756 ms
Total hot run time: 195456 ms

doris-robot · 2025-01-10T18:22:40Z

TeamCity be ut coverage result:
Function Coverage: 39.35% (10255/26059)
Line Coverage: 30.53% (87427/286409)
Region Coverage: 29.55% (44535/150688)
Branch Coverage: 26.11% (22800/87322)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6fe1ca2ba7745ef4f40135925cae1e349038cb9f_6fe1ca2ba7745ef4f40135925cae1e349038cb9f/report/index.html

doris-robot · 2025-01-10T18:27:40Z

ClickBench: Total hot run time: 31.29 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6fe1ca2ba7745ef4f40135925cae1e349038cb9f, data reload: false

query1	0.03	0.04	0.03
query2	0.08	0.03	0.04
query3	0.23	0.07	0.07
query4	1.60	0.11	0.10
query5	0.41	0.41	0.41
query6	1.17	0.65	0.64
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.51	0.52
query10	0.55	0.56	0.57
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.59	0.61
query14	2.84	2.75	2.83
query15	0.90	0.82	0.82
query16	0.38	0.38	0.38
query17	1.05	0.99	1.06
query18	0.23	0.23	0.21
query19	1.89	1.78	1.97
query20	0.02	0.01	0.02
query21	15.36	0.94	0.58
query22	0.77	0.81	0.66
query23	15.25	1.40	0.63
query24	3.27	0.99	1.13
query25	0.23	0.20	0.07
query26	0.27	0.14	0.14
query27	0.07	0.05	0.06
query28	13.79	1.53	1.05
query29	12.59	3.94	3.25
query30	0.24	0.08	0.08
query31	2.82	0.61	0.38
query32	3.22	0.54	0.46
query33	3.11	3.09	3.11
query34	16.63	5.13	4.45
query35	4.44	4.47	4.49
query36	0.65	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.02
query40	0.18	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.15 s
Total hot run time: 31.29 s

github-actions · 2025-01-12T12:08:40Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-01-12T12:08:42Z

PR approved by anyone and no changes requested.

shuke987

LGTM

zhiqiang-hhhh changed the title ~~[DEBUG] CI DEBUG PROFILE~~ [fix][profile] Fix broker load profile Jan 8, 2025

zhiqiang-hhhh force-pushed the debug-profile branch from 968b793 to 01ed312 Compare January 9, 2025 12:30

zhiqiang-hhhh changed the title ~~[fix][profile] Fix broker load profile~~ [chore][profile] LOG enhancement and minor change of profile Jan 9, 2025

zhiqiang-hhhh force-pushed the debug-profile branch from 01ed312 to a85f630 Compare January 9, 2025 12:56

zhiqiang-hhhh force-pushed the debug-profile branch from fcab7bd to f6500eb Compare January 9, 2025 15:41

zhiqiang-hhhh changed the title ~~[chore][profile] LOG enhancement and minor change of profile~~ [chore][profile] log enhancement and minor change of profile Jan 9, 2025

zhiqiang-hhhh marked this pull request as ready for review January 10, 2025 01:50

zhiqiang-hhhh added 4 commits January 10, 2025 09:56

DEV

52e8833

NED

8ada8e3

NED

0f7a39c

DEV

6f8f0f0

zhiqiang-hhhh force-pushed the debug-profile branch from f6500eb to 6f8f0f0 Compare January 10, 2025 02:51

NED

40d87f2

zhiqiang-hhhh changed the title ~~[chore][profile] log enhancement and minor change of profile~~ [refactor][profile] log enhancement and minor change of profile Jan 10, 2025

yiguolei reviewed Jan 10, 2025

View reviewed changes

NEED

6fe1ca2

zhiqiang-hhhh requested a review from yiguolei January 12, 2025 11:50

yiguolei approved these changes Jan 12, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 12, 2025

github-actions bot added the reviewed label Jan 12, 2025

shuke987 approved these changes Jan 12, 2025

View reviewed changes

wangbo merged commit 23cddd0 into apache:master Jan 13, 2025
24 of 26 checks passed

zhiqiang-hhhh deleted the debug-profile branch January 13, 2025 01:57

[refactor][profile] log enhancement and minor change of profile #46564

[refactor][profile] log enhancement and minor change of profile #46564

Conversation

zhiqiang-hhhh commented Jan 7, 2025 • edited Loading

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

hello-stephen commented Jan 7, 2025

zhiqiang-hhhh commented Jan 7, 2025

zhiqiang-hhhh commented Jan 7, 2025

doris-robot commented Jan 7, 2025

doris-robot commented Jan 7, 2025

doris-robot commented Jan 7, 2025

doris-robot commented Jan 7, 2025

zhiqiang-hhhh commented Jan 7, 2025

shuke987 commented Jan 8, 2025

shuke987 commented Jan 8, 2025

shuke987 commented Jan 8, 2025

zhiqiang-hhhh commented Jan 9, 2025

doris-robot commented Jan 9, 2025

zhiqiang-hhhh commented Jan 9, 2025

doris-robot commented Jan 9, 2025

doris-robot commented Jan 9, 2025

doris-robot commented Jan 9, 2025

doris-robot commented Jan 9, 2025

doris-robot commented Jan 10, 2025

zhiqiang-hhhh commented Jan 10, 2025

doris-robot commented Jan 10, 2025

doris-robot commented Jan 10, 2025

doris-robot commented Jan 10, 2025

doris-robot commented Jan 10, 2025

zhiqiang-hhhh commented Jan 10, 2025

doris-robot commented Jan 10, 2025

doris-robot commented Jan 10, 2025

doris-robot commented Jan 10, 2025

doris-robot commented Jan 10, 2025

yiguolei Jan 10, 2025

Choose a reason for hiding this comment

zhiqiang-hhhh Jan 10, 2025

Choose a reason for hiding this comment

yiguolei Jan 10, 2025

Choose a reason for hiding this comment

zhiqiang-hhhh commented Jan 10, 2025

doris-robot commented Jan 10, 2025

doris-robot commented Jan 10, 2025

doris-robot commented Jan 10, 2025

doris-robot commented Jan 10, 2025

github-actions bot commented Jan 12, 2025

github-actions bot commented Jan 12, 2025

shuke987 left a comment

Choose a reason for hiding this comment

zhiqiang-hhhh commented Jan 7, 2025 •

edited

Loading