Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](memory) Fix metadata memory tracking and profile #44739

Merged
merged 3 commits into from
Dec 4, 2024

Conversation

xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Nov 28, 2024

What problem does this PR solve?

Problem Summary:

  1. MemTrackerLimiter and MemoryProfile add metadata and cache type.
  2. metadata memory consists of rowsets, segments, and tablets.
  3. all_segments_mem_tracker excludes the segments in SegmentCache, and all segments use the same memory calculation method.
  4. segments memory is estimated, which is not accurate and can only be used as a reference.
  5. rowsets and tablets only count part of the memory, mainly PB.

TODO, Metadata also has some memory that can be allocated using Doris Allocator.

MemoryProfile:
  MemoryOverviewSnapshot:
     - PhysicalMemory(VmRSS) Current: 5.85 GB (Peak: 7.13 GB)
     - VirtualMemory(VmSize) Current: 61.35 GB (Peak: 61.67 GB)
    UntrackedMemory:
       - Memory Current: 3.61 GB (Peak: 3.77 GB)
    TrackedMemory:
       - Memory Current: 2.24 GB (Peak: 3.48 GB)
      TasksMemory:
         - Memory Current: 0 (Peak: 749.17 MB)
           - ReservedMemory Current: 0 (Peak: 0)
        Details:
           - Compaction Current: 0 (Peak: 13.60 MB)
           - Load Current: 0 (Peak: 749.17 MB)
             - AllMemTablesMemory Current: 0 (Peak: 853.20 MB)
           - Other Current: 0 (Peak: 0)
           - Query Current: 0 (Peak: 65.20 MB)
           - SchemaChange Current: 0 (Peak: 0)
      GlobalMemory:
         - Memory Current: 55.03 MB (Peak: 70.39 MB)
      MetadataMemory:
         - Memory Current: 972.13 MB (Peak: 972.13 MB)
      CacheMemory:
         - Memory Current: 360.21 MB (Peak: 675.32 MB)
      JemallocMemory:
         - Memory Current: 905.55 MB (Peak: 1.80 GB)
        Details:
           - Cache Current: 646.47 MB (Peak: 1.59 GB)
           - Metadata Current: 259.08 MB (Peak: 259.72 MB)
  GlobalMemorySnapshot:
    Orphan@global@id=3144f940fa84f930-d82d459289789494:
       - Memory Current: 1.73 MB (Peak: 1.73 MB)
    IOBufBlockMemory@global@id=1f4316bc5fb4c3d4-73f14d3652ed269b:
       - Memory Current: 2.65 GB (Peak: 2.65 GB)
    PointQueryExecutor@global@id=794878deaa4d8f56-769fbc87ccb7ec8d:
       - Memory Current: 36.90 MB (Peak: 36.90 MB)
    BlockCompression@global@id=0a4d429b1915dd8b-c2e429b1186d69a8:
       - Memory Current: 563.01 MB (Peak: 563.01 MB)
    RowIdStorageReader@global@id=a546ca5878c9204e-c4c72ea9088ca9a0:
       - Memory Current: 101.25 KB (Peak: 101.25 KB)
    SubcolumnsTree@global@id=294bde93c01d4ba6-4ce7b3290489a1b8:
       - Memory Current: 95.61 MB (Peak: 95.61 MB)
    S3FileBuffer@global@id=bc437d624d9a234d-03eb4bbc3d030a99:
       - Memory Current: 0 (Peak: 0)
  MetadataMemorySnapshot:
    Tablets(not in SchemaCache, TabletSchemaCache)@metadata@id=7646d676ad9f8de9-98beb5f1a8aa739b:
       - Memory Current: 26.84 GB (Peak: 26.84 GB)
    Segments(not in SegmentCache)@metadata@id=864b4887722b327a-516c912d4602b59b:
       - Memory Current: 4.98 MB (Peak: 4.98 MB)
    Rowsets@metadata@id=f0440aa247730497-752a527c71e3c393:
       - Memory Current: 5.57 GB (Peak: 5.57 GB)
    SegmentCache[size]@metadata@id=b34e798db2c25109-04c7ff3c722e4cbb:
       - Memory Current: 4.86 GB (Peak: 4.86 GB)
    SchemaCache[number]@metadata@id=9f41f61531da3fd9-163834a6a6f83eba:
       - Memory Current: 103.38 MB (Peak: 103.38 MB)
    TabletSchemaCache[number]@metadata@id=f9475c16ba7ea1ab-1d28b38568ef0b8d:
       - Memory Current: 170.65 MB (Peak: 170.65 MB)
  CacheMemorySnapshot:
    QueryCache@cache@id=504af426a68103cf-a60f36a48bc1998a:
       - Memory Current: 0 (Peak: 0)
    DataPageCache[size]@cache@id=f744bb4ef53c307b-4c021a8f9b8a77a5:
       - Memory Current: 14.33 GB (Peak: 14.33 GB)
    IndexPageCache[size]@cache@id=cb4ed0e456f081d4-d635edda9aa2778b:
       - Memory Current: 5.25 GB (Peak: 5.25 GB)
    PKIndexPageCache[size]@cache@id=8e4bf12b4d39133a-e0d878a18d73619b:
       - Memory Current: 2.72 GB (Peak: 2.72 GB)
    PointQueryRowCache[size]@cache@id=024dc624c80205f1-ca3221a154e4b68a:
       - Memory Current: 94.84 KB (Peak: 94.84 KB)
    CommonObjLRUCache[number]@cache@id=af40d4a03b391540-43338381c8e826a4:
       - Memory Current: 0 (Peak: 0)
    PointQueryLookupConnectionCache[number]@cache@id=284fa848dd97ccc8-56c61386a4950784:
       - Memory Current: 337.97 KB (Peak: 337.97 KB)
    InvertedIndexSearcherCache[size]@cache@id=f5404416ad0a9465-3cf12f5224276d8a:
       - Memory Current: 111.52 MB (Peak: 111.52 MB)
    InvertedIndexQueryCache[size]@cache@id=6f4d3c55508e3531-3fafe328c6d3e1ab:
       - Memory Current: 51.97 MB (Peak: 51.97 MB)
    QueryCache[size]@cache@id=274d47670e0796ad-542ad958334ff988:
       - Memory Current: 0 (Peak: 0)
    LastSuccessChannelCache[size]@cache@id=08449badfd6abb81-5b30599bfc60f79f:
       - Memory Current: 0 (Peak: 0)
    TabletColumnObjectPool[number]@cache@id=7b42da8391945265-06697f83a6bf01a7:
       - Memory Current: 151.67 MB (Peak: 151.67 MB)
    MowTabletVersionCache[number]@cache@id=f345a1adbd2a87dd-74b9caa3b30cf18b:
       - Memory Current: 150.94 MB (Peak: 150.94 MB)
    CreateTabletRRIdxCache[number]@cache@id=f043eee91e9aef2d-ef3a2245d50545ae:
       - Memory Current: 15.20 MB (Peak: 15.20 MB)
    MowDeleteBitmapAggCache[size]@cache@id=df47a8ed5bf3f11f-ac944d7d7e79d4b3:
       - Memory Current: 181.11 MB (Peak: 181.11 MB)
  TopMemoryTasksSnapshot:
    SnapshotManager@other_task@id=11480ad6017368b9-8aef2150427d229f:
       - Memory Current: 0 (Peak: 0)
    StreamLoadPipe@load@id=024958077445f4f7-d739fd5231dca890:
       - Memory Current: 1.39 GB (Peak: 1.39 GB)
    SegCompaction@compaction@id=2d4d5c796a9f48ee-fdf5dd9f8faf83a8:
       - Memory Current: 0 (Peak: 0)
    Load#Id=544468c7f5a9bac6-2c78fc9962f3908d@load@id=52497774c12c5ad1-1fda35667a3d0e8b:
       - Limit: 2.00 GB
       - Memory Current: 6.06 MB (Peak: 6.06 MB)
    Load#Id=624e7947099ef183-577a22cca9d34d89@load@id=ed4691ce5a748ca8-e49ef7cc9be10b95:
       - Limit: 2.00 GB
       - Memory Current: 6.06 MB (Peak: 6.06 MB)

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xinyiZzz xinyiZzz force-pushed the 20241126_fix_memory_metadata2 branch from 09d18cb to bf3fb24 Compare November 28, 2024 10:18
@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40044 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bf3fb248c476f6f982a3aa52e28509d7e1e3b02f, data reload: false

------ Round 1 ----------------------------------
q1	17601	7594	7310	7310
q2	2036	174	172	172
q3	10545	1165	1203	1165
q4	10220	773	696	696
q5	7607	2778	2707	2707
q6	245	148	147	147
q7	1013	622	609	609
q8	9242	1899	2013	1899
q9	6687	6530	6521	6521
q10	6970	2301	2281	2281
q11	449	250	261	250
q12	418	215	218	215
q13	17765	2999	3002	2999
q14	239	217	217	217
q15	574	536	526	526
q16	653	591	579	579
q17	981	569	523	523
q18	7387	6620	6660	6620
q19	1346	1039	969	969
q20	488	176	179	176
q21	4011	3154	3260	3154
q22	393	316	309	309
Total cold run time: 106870 ms
Total hot run time: 40044 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7216	7207	7271	7207
q2	324	232	230	230
q3	2906	2786	3093	2786
q4	2107	1815	1807	1807
q5	5586	5774	5641	5641
q6	221	138	139	138
q7	2204	1747	1801	1747
q8	3384	3608	3488	3488
q9	8980	8982	9159	8982
q10	3608	3564	3554	3554
q11	615	502	493	493
q12	818	622	617	617
q13	11590	3203	3118	3118
q14	308	274	270	270
q15	568	510	536	510
q16	709	649	623	623
q17	1796	1587	1580	1580
q18	7856	7430	7465	7430
q19	1672	1475	1605	1475
q20	2053	1823	1808	1808
q21	5428	5349	5225	5225
q22	665	516	563	516
Total cold run time: 70614 ms
Total hot run time: 59245 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191661 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bf3fb248c476f6f982a3aa52e28509d7e1e3b02f, data reload: false

query1	970	382	361	361
query2	6506	2149	2087	2087
query3	6704	208	212	208
query4	34207	23505	23682	23505
query5	4371	472	467	467
query6	274	180	174	174
query7	4615	306	311	306
query8	303	230	229	229
query9	9593	2786	2779	2779
query10	466	256	253	253
query11	18011	15225	15240	15225
query12	163	100	108	100
query13	1634	421	397	397
query14	9667	6776	7668	6776
query15	289	179	183	179
query16	8185	471	481	471
query17	1738	562	535	535
query18	2124	303	285	285
query19	365	148	152	148
query20	125	114	114	114
query21	203	100	101	100
query22	4347	4302	4247	4247
query23	34957	34754	34332	34332
query24	11231	2464	2562	2464
query25	704	440	390	390
query26	1819	154	153	153
query27	2896	282	279	279
query28	8200	2470	2471	2470
query29	1017	413	423	413
query30	299	154	149	149
query31	1021	794	846	794
query32	99	56	59	56
query33	763	317	320	317
query34	928	507	527	507
query35	863	744	716	716
query36	1100	940	963	940
query37	161	71	71	71
query38	4328	4230	4272	4230
query39	1488	1415	1420	1415
query40	282	104	100	100
query41	49	43	42	42
query42	112	100	102	100
query43	536	500	499	499
query44	1334	819	819	819
query45	182	164	164	164
query46	1146	717	705	705
query47	1919	1842	1852	1842
query48	415	307	316	307
query49	1218	397	380	380
query50	794	396	392	392
query51	7417	7245	7153	7153
query52	103	94	92	92
query53	258	188	185	185
query54	1108	448	436	436
query55	79	74	77	74
query56	268	270	248	248
query57	1306	1164	1135	1135
query58	242	237	260	237
query59	3204	3059	2814	2814
query60	289	274	271	271
query61	123	154	112	112
query62	827	681	688	681
query63	218	187	189	187
query64	5199	663	640	640
query65	3292	3237	3212	3212
query66	1195	313	305	305
query67	16197	15779	15652	15652
query68	4976	563	582	563
query69	405	263	266	263
query70	1191	1148	1115	1115
query71	323	277	263	263
query72	6433	4113	4052	4052
query73	844	358	370	358
query74	10007	9011	8932	8932
query75	3406	2667	2697	2667
query76	2744	1083	1062	1062
query77	424	285	272	272
query78	10289	9390	9385	9385
query79	2445	609	622	609
query80	1126	454	439	439
query81	530	245	227	227
query82	937	121	115	115
query83	244	147	141	141
query84	228	72	75	72
query85	1396	304	297	297
query86	453	309	302	302
query87	4654	4502	4717	4502
query88	4136	2248	2215	2215
query89	412	300	288	288
query90	1945	226	221	221
query91	137	102	106	102
query92	67	56	55	55
query93	1609	565	558	558
query94	737	338	331	331
query95	410	327	331	327
query96	610	286	278	278
query97	2840	2691	2684	2684
query98	217	196	198	196
query99	1573	1364	1344	1344
Total cold run time: 303794 ms
Total hot run time: 191661 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.1 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bf3fb248c476f6f982a3aa52e28509d7e1e3b02f, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.42	0.42	0.42
query6	1.17	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.51	0.53
query10	0.56	0.55	0.55
query11	0.13	0.11	0.11
query12	0.13	0.11	0.11
query13	0.62	0.59	0.60
query14	2.70	2.83	2.84
query15	0.90	0.83	0.83
query16	0.39	0.39	0.38
query17	1.06	1.06	1.04
query18	0.22	0.20	0.20
query19	1.90	1.80	2.03
query20	0.02	0.01	0.01
query21	15.38	0.60	0.60
query22	2.54	3.02	1.79
query23	17.06	1.01	0.92
query24	2.50	1.14	1.27
query25	0.27	0.15	0.13
query26	0.37	0.14	0.13
query27	0.03	0.04	0.05
query28	10.63	1.10	1.07
query29	12.55	3.24	3.25
query30	0.25	0.06	0.08
query31	2.85	0.38	0.40
query32	3.27	0.47	0.47
query33	2.99	3.02	3.05
query34	17.14	4.54	4.51
query35	4.64	4.51	4.49
query36	0.66	0.49	0.49
query37	0.09	0.07	0.06
query38	0.04	0.04	0.04
query39	0.04	0.02	0.03
query40	0.17	0.12	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.04
Total cold run time: 106.43 s
Total hot run time: 33.1 s

@xinyiZzz xinyiZzz force-pushed the 20241126_fix_memory_metadata2 branch from bf3fb24 to db61f10 Compare November 29, 2024 11:49
@yiguolei
Copy link
Contributor

yiguolei commented Dec 1, 2024

run buildall

yiguolei
yiguolei previously approved these changes Dec 1, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 1, 2024
Copy link
Contributor

github-actions bot commented Dec 1, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Dec 1, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 40089 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit db61f1073998b0f61014670a60b4c8de1c4bf065, data reload: false

------ Round 1 ----------------------------------
q1	17625	7546	7337	7337
q2	2039	175	166	166
q3	10558	1097	1154	1097
q4	10553	709	724	709
q5	7594	2724	2702	2702
q6	244	146	145	145
q7	995	618	597	597
q8	9256	1889	1992	1889
q9	6672	6533	6476	6476
q10	6998	2320	2338	2320
q11	475	274	270	270
q12	419	221	232	221
q13	17781	3032	3057	3032
q14	237	212	220	212
q15	586	539	524	524
q16	641	589	593	589
q17	989	501	546	501
q18	7305	6668	6788	6668
q19	1337	1027	986	986
q20	484	183	181	181
q21	4206	3295	3145	3145
q22	371	322	323	322
Total cold run time: 107365 ms
Total hot run time: 40089 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7218	7446	7231	7231
q2	331	231	243	231
q3	2952	2870	2944	2870
q4	2083	1811	1859	1811
q5	5645	5697	5656	5656
q6	228	138	143	138
q7	2265	1856	1901	1856
q8	3427	3549	3588	3549
q9	9120	9150	9137	9137
q10	3627	3581	3577	3577
q11	592	504	517	504
q12	808	614	607	607
q13	11890	3253	3238	3238
q14	304	288	272	272
q15	585	534	517	517
q16	688	647	644	644
q17	1879	1642	1604	1604
q18	8433	7845	7634	7634
q19	1733	1556	1677	1556
q20	2147	1884	1881	1881
q21	5582	5520	5486	5486
q22	622	555	582	555
Total cold run time: 72159 ms
Total hot run time: 60554 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197943 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit db61f1073998b0f61014670a60b4c8de1c4bf065, data reload: false

query1	1239	962	948	948
query2	6254	2114	2087	2087
query3	11144	4680	4548	4548
query4	39740	23923	23667	23667
query5	4517	492	450	450
query6	245	189	179	179
query7	3671	310	312	310
query8	315	227	239	227
query9	6381	2629	2620	2620
query10	408	255	245	245
query11	15814	15246	15298	15246
query12	156	110	101	101
query13	854	397	419	397
query14	9433	7159	7670	7159
query15	234	193	190	190
query16	7899	461	496	461
query17	1531	565	572	565
query18	2024	306	309	306
query19	355	158	155	155
query20	118	115	110	110
query21	206	108	110	108
query22	4716	4737	4624	4624
query23	35339	34484	34526	34484
query24	10543	2623	2470	2470
query25	561	405	379	379
query26	1157	156	152	152
query27	2636	278	289	278
query28	7554	2461	2484	2461
query29	684	412	423	412
query30	225	150	145	145
query31	999	833	839	833
query32	67	59	56	56
query33	634	312	311	311
query34	939	502	528	502
query35	897	793	769	769
query36	1098	961	974	961
query37	137	79	75	75
query38	4553	4441	4413	4413
query39	1527	1458	1484	1458
query40	202	105	109	105
query41	46	51	45	45
query42	127	107	104	104
query43	539	508	483	483
query44	1298	817	847	817
query45	200	182	177	177
query46	1211	724	744	724
query47	2035	1955	1952	1952
query48	416	322	330	322
query49	955	404	388	388
query50	853	389	392	389
query51	7319	7186	7386	7186
query52	104	91	91	91
query53	271	186	185	185
query54	531	395	395	395
query55	74	76	74	74
query56	272	236	225	225
query57	1227	1130	1134	1130
query58	223	212	205	205
query59	3156	3174	3087	3087
query60	260	241	248	241
query61	109	109	105	105
query62	862	656	681	656
query63	214	181	183	181
query64	3476	674	650	650
query65	3302	3268	3196	3196
query66	774	302	304	302
query67	15956	15604	15545	15545
query68	4344	572	579	572
query69	465	254	251	251
query70	1240	1149	1093	1093
query71	426	253	259	253
query72	6564	4129	4214	4129
query73	782	358	352	352
query74	9963	9236	8995	8995
query75	3439	2746	2667	2667
query76	3108	1093	1143	1093
query77	624	293	312	293
query78	10382	9350	9361	9350
query79	1772	590	593	590
query80	1118	437	418	418
query81	566	226	224	224
query82	358	126	122	122
query83	238	141	145	141
query84	247	69	68	68
query85	1427	303	303	303
query86	369	299	298	298
query87	4857	4643	4700	4643
query88	3750	2165	2093	2093
query89	433	301	301	301
query90	2002	193	185	185
query91	150	102	101	101
query92	58	49	49	49
query93	2759	561	563	561
query94	947	298	290	290
query95	360	249	254	249
query96	643	281	288	281
query97	2840	2651	2697	2651
query98	215	195	201	195
query99	1530	1310	1348	1310
Total cold run time: 300792 ms
Total hot run time: 197943 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.76 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit db61f1073998b0f61014670a60b4c8de1c4bf065, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.23	0.08	0.06
query4	1.63	0.10	0.10
query5	0.44	0.41	0.40
query6	1.16	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.52	0.50
query10	0.55	0.56	0.56
query11	0.14	0.10	0.10
query12	0.15	0.12	0.11
query13	0.60	0.61	0.61
query14	2.83	2.85	2.75
query15	0.90	0.82	0.82
query16	0.39	0.37	0.37
query17	1.09	1.08	1.03
query18	0.23	0.21	0.20
query19	1.91	1.86	2.02
query20	0.02	0.00	0.02
query21	15.36	0.60	0.57
query22	2.89	2.00	1.80
query23	17.08	1.30	0.73
query24	2.50	1.24	1.51
query25	0.19	0.24	0.05
query26	0.51	0.14	0.14
query27	0.04	0.04	0.05
query28	10.48	1.10	1.06
query29	12.52	3.30	3.26
query30	0.25	0.06	0.06
query31	2.87	0.38	0.37
query32	3.30	0.47	0.47
query33	3.02	3.12	3.01
query34	17.08	4.47	4.47
query35	4.56	4.48	4.49
query36	0.65	0.48	0.48
query37	0.09	0.06	0.05
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.81 s
Total hot run time: 32.76 s

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Dec 1, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 1, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 40012 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d44aa87573bbc64d0ec035223d36e4ee256ae48f, data reload: false

------ Round 1 ----------------------------------
q1	17838	7561	7291	7291
q2	2044	186	173	173
q3	10679	1118	1190	1118
q4	10513	726	769	726
q5	7602	2715	2712	2712
q6	237	145	143	143
q7	1019	594	604	594
q8	9239	1904	1946	1904
q9	6755	6510	6462	6462
q10	6973	2290	2299	2290
q11	457	268	254	254
q12	449	216	215	215
q13	17760	3031	3004	3004
q14	246	217	213	213
q15	576	521	541	521
q16	638	586	589	586
q17	981	602	527	527
q18	7333	6620	6701	6620
q19	1329	921	1045	921
q20	478	180	180	180
q21	4055	3283	3239	3239
q22	378	322	319	319
Total cold run time: 107579 ms
Total hot run time: 40012 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7263	7224	7238	7224
q2	328	230	226	226
q3	2913	2827	2909	2827
q4	2091	1825	1858	1825
q5	5661	5671	5653	5653
q6	227	138	144	138
q7	2216	1787	1804	1787
q8	3418	3507	3480	3480
q9	9067	9061	9069	9061
q10	3595	3563	3540	3540
q11	613	519	525	519
q12	814	630	585	585
q13	10605	3206	3254	3206
q14	317	280	286	280
q15	582	519	519	519
q16	677	653	637	637
q17	1837	1641	1592	1592
q18	8396	7681	7646	7646
q19	1691	1697	1625	1625
q20	2132	1875	1873	1873
q21	5697	5358	5499	5358
q22	662	589	628	589
Total cold run time: 70802 ms
Total hot run time: 60190 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195967 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d44aa87573bbc64d0ec035223d36e4ee256ae48f, data reload: false

query1	1243	957	986	957
query2	6229	2068	2030	2030
query3	11069	4551	4550	4550
query4	67155	28308	23591	23591
query5	4950	456	459	456
query6	407	181	195	181
query7	5696	310	280	280
query8	307	236	225	225
query9	9390	2609	2588	2588
query10	469	256	253	253
query11	17659	15349	15882	15349
query12	162	99	104	99
query13	1606	406	420	406
query14	9993	6652	7154	6652
query15	214	190	184	184
query16	7483	522	432	432
query17	1057	600	604	600
query18	1887	305	304	304
query19	264	158	156	156
query20	115	115	110	110
query21	201	105	104	104
query22	4656	4517	4532	4517
query23	34815	34392	34787	34392
query24	5569	2469	2447	2447
query25	491	382	393	382
query26	641	151	149	149
query27	2015	275	290	275
query28	4416	2476	2439	2439
query29	679	412	406	406
query30	205	156	143	143
query31	1024	826	839	826
query32	65	51	54	51
query33	427	288	308	288
query34	921	513	528	513
query35	888	741	755	741
query36	1091	976	984	976
query37	139	71	71	71
query38	4438	4315	4416	4315
query39	1500	1504	1463	1463
query40	199	99	96	96
query41	42	40	42	40
query42	111	94	99	94
query43	541	479	482	479
query44	1181	836	810	810
query45	187	161	177	161
query46	1174	736	711	711
query47	2006	1930	1880	1880
query48	424	302	310	302
query49	719	378	413	378
query50	846	390	398	390
query51	7437	7197	7032	7032
query52	96	86	87	86
query53	247	172	184	172
query54	506	386	395	386
query55	78	72	76	72
query56	256	221	251	221
query57	1254	1079	1119	1079
query58	221	208	200	200
query59	3166	3106	2831	2831
query60	257	234	225	225
query61	106	100	98	98
query62	796	676	683	676
query63	210	181	187	181
query64	1376	663	668	663
query65	3238	3231	3186	3186
query66	627	297	300	297
query67	15856	15432	15639	15432
query68	4372	561	563	561
query69	409	245	247	245
query70	1188	1139	1120	1120
query71	317	253	272	253
query72	6445	4003	3993	3993
query73	752	353	356	353
query74	9952	8955	9051	8955
query75	3383	2663	2614	2614
query76	1834	1091	1144	1091
query77	477	266	260	260
query78	10552	9540	9460	9460
query79	1414	604	603	603
query80	869	426	490	426
query81	492	233	227	227
query82	1302	118	113	113
query83	201	142	136	136
query84	284	68	74	68
query85	875	292	287	287
query86	331	300	302	300
query87	4780	4575	4604	4575
query88	3573	2172	2125	2125
query89	419	296	294	294
query90	1901	181	178	178
query91	139	105	99	99
query92	64	47	48	47
query93	1939	546	556	546
query94	727	276	300	276
query95	352	248	241	241
query96	612	277	271	271
query97	2840	2685	2680	2680
query98	223	192	197	192
query99	1716	1310	1299	1299
Total cold run time: 319427 ms
Total hot run time: 195967 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.27 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d44aa87573bbc64d0ec035223d36e4ee256ae48f, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.07
query4	1.63	0.10	0.10
query5	0.44	0.40	0.42
query6	1.15	0.66	0.64
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.56	0.53	0.49
query10	0.57	0.57	0.56
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.61	0.61	0.61
query14	2.71	2.85	2.85
query15	0.89	0.84	0.82
query16	0.38	0.39	0.38
query17	1.02	0.99	1.07
query18	0.22	0.19	0.20
query19	2.00	1.83	2.09
query20	0.01	0.02	0.01
query21	15.36	0.58	0.58
query22	2.65	2.04	2.67
query23	16.96	1.01	0.77
query24	3.30	1.56	1.40
query25	0.24	0.08	0.16
query26	0.48	0.13	0.14
query27	0.04	0.04	0.04
query28	9.76	1.12	1.08
query29	12.54	3.23	3.19
query30	0.25	0.06	0.06
query31	2.86	0.37	0.38
query32	3.28	0.46	0.47
query33	3.02	3.01	3.03
query34	17.01	4.49	4.50
query35	4.55	4.59	4.56
query36	0.67	0.49	0.47
query37	0.09	0.05	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.03
query40	0.17	0.13	0.12
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.29 s
Total hot run time: 33.27 s

@xinyiZzz xinyiZzz force-pushed the 20241126_fix_memory_metadata2 branch from d44aa87 to 0531b4a Compare December 2, 2024 02:52
@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Dec 2, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40256 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0531b4a52c8eedfef9b77a7d01393e0d96de1dea, data reload: false

------ Round 1 ----------------------------------
q1	17587	7472	7378	7378
q2	2045	173	164	164
q3	10606	1099	1222	1099
q4	10574	783	746	746
q5	7613	2752	2732	2732
q6	244	147	146	146
q7	1004	645	623	623
q8	9255	1883	1956	1883
q9	6720	6552	6603	6552
q10	7007	2279	2329	2279
q11	454	259	252	252
q12	421	224	219	219
q13	17769	3008	3046	3008
q14	239	222	209	209
q15	582	552	531	531
q16	646	584	577	577
q17	988	568	493	493
q18	7366	6622	6828	6622
q19	1342	1035	1046	1035
q20	475	177	183	177
q21	4119	3280	3218	3218
q22	389	325	313	313
Total cold run time: 107445 ms
Total hot run time: 40256 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7315	7311	7209	7209
q2	342	233	237	233
q3	2917	2818	2993	2818
q4	2052	1814	1848	1814
q5	5634	5716	5679	5679
q6	233	148	145	145
q7	2227	1856	1815	1815
q8	3448	3608	3565	3565
q9	9058	9073	9124	9073
q10	3618	3604	3526	3526
q11	598	504	515	504
q12	816	624	593	593
q13	11450	3241	3212	3212
q14	314	291	274	274
q15	573	534	521	521
q16	713	658	627	627
q17	1895	1673	1624	1624
q18	8501	7763	7566	7566
q19	1783	1674	1646	1646
q20	2111	1910	1859	1859
q21	5807	5403	5558	5403
q22	662	583	568	568
Total cold run time: 72067 ms
Total hot run time: 60274 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197723 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0531b4a52c8eedfef9b77a7d01393e0d96de1dea, data reload: false

query1	1249	939	944	939
query2	6234	2142	2062	2062
query3	10963	4389	4493	4389
query4	67375	33988	23818	23818
query5	5134	456	453	453
query6	334	199	174	174
query7	5000	310	302	302
query8	295	240	242	240
query9	6550	2651	2646	2646
query10	403	238	249	238
query11	15491	15451	15873	15451
query12	143	98	109	98
query13	1045	394	386	386
query14	10612	7550	7247	7247
query15	218	190	202	190
query16	7066	503	449	449
query17	1279	596	621	596
query18	1747	326	326	326
query19	212	177	178	177
query20	121	111	118	111
query21	223	105	106	105
query22	5003	4735	4800	4735
query23	34593	34400	34494	34400
query24	5416	2512	2389	2389
query25	472	381	395	381
query26	622	151	147	147
query27	1885	278	280	278
query28	4084	2473	2477	2473
query29	670	414	412	412
query30	207	143	150	143
query31	1041	910	838	838
query32	70	56	53	53
query33	389	284	286	284
query34	929	522	532	522
query35	895	751	740	740
query36	1090	951	971	951
query37	120	77	75	75
query38	4489	4362	4491	4362
query39	1538	1484	1476	1476
query40	202	98	100	98
query41	43	46	42	42
query42	104	98	98	98
query43	534	495	493	493
query44	1214	855	820	820
query45	195	169	172	169
query46	1186	727	752	727
query47	2031	1940	1965	1940
query48	407	312	323	312
query49	700	380	401	380
query50	867	392	388	388
query51	7319	7286	7128	7128
query52	95	86	85	85
query53	251	186	172	172
query54	507	396	389	389
query55	74	72	77	72
query56	240	238	233	233
query57	1244	1117	1133	1117
query58	218	201	204	201
query59	3139	3056	2973	2973
query60	297	244	230	230
query61	105	108	111	108
query62	792	686	674	674
query63	210	180	184	180
query64	1356	665	638	638
query65	3254	3179	3169	3169
query66	696	297	352	297
query67	16025	15924	15640	15640
query68	3905	571	570	570
query69	409	257	246	246
query70	1185	1076	1107	1076
query71	385	241	244	241
query72	6522	4047	3999	3999
query73	784	368	360	360
query74	10081	8926	9063	8926
query75	3405	2656	2668	2656
query76	1603	974	1091	974
query77	488	264	261	261
query78	10529	9515	9480	9480
query79	1141	590	602	590
query80	771	417	473	417
query81	507	226	237	226
query82	240	127	118	118
query83	199	146	159	146
query84	274	80	68	68
query85	1010	293	288	288
query86	357	305	295	295
query87	4792	4676	4647	4647
query88	3440	2171	2134	2134
query89	400	304	339	304
query90	2079	183	184	183
query91	139	103	105	103
query92	63	48	55	48
query93	1343	542	547	542
query94	799	305	293	293
query95	345	243	254	243
query96	609	283	287	283
query97	2869	2683	2675	2675
query98	223	192	208	192
query99	1762	1310	1342	1310
Total cold run time: 311170 ms
Total hot run time: 197723 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.13 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0531b4a52c8eedfef9b77a7d01393e0d96de1dea, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.24	0.07	0.08
query4	1.61	0.11	0.10
query5	0.44	0.43	0.42
query6	1.16	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.50	0.50
query10	0.54	0.55	0.55
query11	0.13	0.10	0.10
query12	0.14	0.10	0.12
query13	0.61	0.60	0.60
query14	2.72	2.84	2.72
query15	0.90	0.82	0.83
query16	0.39	0.40	0.39
query17	1.05	1.07	1.06
query18	0.23	0.21	0.22
query19	1.95	1.86	1.99
query20	0.01	0.01	0.01
query21	15.38	0.59	0.58
query22	2.40	2.88	1.92
query23	17.14	0.89	0.65
query24	2.84	1.37	1.67
query25	0.26	0.08	0.13
query26	0.65	0.14	0.14
query27	0.05	0.03	0.04
query28	10.08	1.09	1.08
query29	12.57	3.23	3.25
query30	0.25	0.06	0.06
query31	2.87	0.39	0.39
query32	3.29	0.46	0.47
query33	3.04	3.16	3.15
query34	17.20	4.46	4.45
query35	4.54	4.51	4.49
query36	0.66	0.48	0.49
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.09	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.56 s
Total hot run time: 33.13 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.37% (9991/26036)
Line Coverage: 29.48% (83767/284189)
Region Coverage: 28.56% (43039/150680)
Branch Coverage: 25.18% (21879/86876)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0531b4a52c8eedfef9b77a7d01393e0d96de1dea_0531b4a52c8eedfef9b77a7d01393e0d96de1dea/report/index.html

be/src/common/daemon.cpp Outdated Show resolved Hide resolved
@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Dec 3, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40352 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2a3783226bec2ce4d1499526b5055e87df47de98, data reload: false

------ Round 1 ----------------------------------
q1	17684	7555	7318	7318
q2	2046	169	186	169
q3	10623	1170	1198	1170
q4	10345	755	730	730
q5	7606	2761	2725	2725
q6	239	146	149	146
q7	1021	624	612	612
q8	9266	1907	1947	1907
q9	6741	6579	6534	6534
q10	7000	2297	2298	2297
q11	466	249	251	249
q12	434	217	226	217
q13	17796	3000	3018	3000
q14	237	215	231	215
q15	568	540	528	528
q16	637	580	590	580
q17	986	534	522	522
q18	7542	6658	6804	6658
q19	1322	1076	1047	1047
q20	487	177	183	177
q21	4058	3392	3238	3238
q22	372	313	324	313
Total cold run time: 107476 ms
Total hot run time: 40352 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7269	7234	7263	7234
q2	334	227	231	227
q3	2921	2884	3043	2884
q4	2130	1846	1772	1772
q5	5585	5722	5672	5672
q6	220	137	141	137
q7	2211	1779	1786	1779
q8	3380	3531	3533	3531
q9	9115	9131	9095	9095
q10	3613	3538	3536	3536
q11	598	520	489	489
q12	802	613	633	613
q13	10626	3145	3226	3145
q14	282	273	275	273
q15	566	510	529	510
q16	672	638	625	625
q17	1830	1583	1581	1581
q18	7768	7513	7280	7280
q19	1679	1510	1531	1510
q20	2045	1800	1837	1800
q21	5393	5250	5234	5234
q22	610	536	545	536
Total cold run time: 69649 ms
Total hot run time: 59463 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190672 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2a3783226bec2ce4d1499526b5055e87df47de98, data reload: false

query1	953	394	374	374
query2	7401	2164	2027	2027
query3	6705	215	216	215
query4	34245	23709	23487	23487
query5	4341	447	451	447
query6	283	206	170	170
query7	4616	302	302	302
query8	304	243	248	243
query9	9809	2720	2714	2714
query10	474	258	265	258
query11	18189	15235	15383	15235
query12	163	104	103	103
query13	1646	396	381	381
query14	10154	6463	6840	6463
query15	259	174	195	174
query16	8105	441	456	441
query17	1641	552	546	546
query18	2119	277	287	277
query19	371	148	171	148
query20	117	111	108	108
query21	209	105	98	98
query22	4684	4431	4496	4431
query23	35074	34231	34341	34231
query24	10509	2398	2500	2398
query25	631	375	382	375
query26	1473	142	151	142
query27	2867	272	276	272
query28	8197	2463	2455	2455
query29	919	395	395	395
query30	291	149	144	144
query31	1033	816	832	816
query32	90	57	56	56
query33	766	293	282	282
query34	966	497	524	497
query35	890	747	743	743
query36	1094	944	951	944
query37	149	74	73	73
query38	4354	4259	4155	4155
query39	1487	1415	1409	1409
query40	305	104	98	98
query41	45	43	43	43
query42	113	98	100	98
query43	539	493	492	492
query44	1328	829	811	811
query45	189	166	169	166
query46	1158	690	705	690
query47	1969	1875	1882	1875
query48	416	308	313	308
query49	1179	377	390	377
query50	814	368	378	368
query51	7303	7167	7068	7068
query52	95	88	89	88
query53	252	186	180	180
query54	1281	405	411	405
query55	80	78	79	78
query56	281	231	276	231
query57	1274	1123	1141	1123
query58	226	207	214	207
query59	3203	3041	3065	3041
query60	273	246	243	243
query61	106	102	105	102
query62	892	674	667	667
query63	211	179	183	179
query64	5195	661	692	661
query65	3269	3178	3215	3178
query66	1437	308	305	305
query67	15976	15580	15618	15580
query68	4062	616	541	541
query69	402	243	247	243
query70	1190	1127	1126	1126
query71	325	258	245	245
query72	6214	4108	4067	4067
query73	766	358	357	357
query74	9824	9042	8942	8942
query75	3453	2661	2691	2661
query76	2674	1097	1030	1030
query77	390	263	267	263
query78	10282	9514	9408	9408
query79	1119	598	584	584
query80	814	524	451	451
query81	479	226	234	226
query82	1266	114	119	114
query83	265	149	152	149
query84	229	72	67	67
query85	1057	293	304	293
query86	365	302	303	302
query87	4739	4655	4564	4564
query88	3417	2189	2217	2189
query89	405	296	298	296
query90	2261	191	183	183
query91	142	111	101	101
query92	60	52	52	52
query93	1054	533	529	529
query94	904	290	285	285
query95	360	241	245	241
query96	601	279	275	275
query97	2838	2689	2691	2689
query98	214	198	195	195
query99	1533	1322	1313	1313
Total cold run time: 300691 ms
Total hot run time: 190672 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.37% (9979/26004)
Line Coverage: 29.50% (83796/284094)
Region Coverage: 28.60% (43063/150582)
Branch Coverage: 25.22% (21903/86856)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2a3783226bec2ce4d1499526b5055e87df47de98_2a3783226bec2ce4d1499526b5055e87df47de98/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 32.64 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2a3783226bec2ce4d1499526b5055e87df47de98, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.63	0.10	0.11
query5	0.43	0.39	0.42
query6	1.16	0.67	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.54	0.50
query10	0.54	0.55	0.56
query11	0.14	0.10	0.11
query12	0.14	0.10	0.11
query13	0.61	0.60	0.60
query14	2.74	2.75	2.85
query15	0.92	0.84	0.83
query16	0.37	0.38	0.38
query17	1.07	1.06	1.04
query18	0.23	0.21	0.22
query19	1.93	1.75	2.02
query20	0.02	0.01	0.01
query21	15.36	0.60	0.58
query22	2.54	2.77	1.67
query23	16.99	1.00	0.88
query24	2.86	1.22	1.82
query25	0.36	0.18	0.06
query26	0.40	0.14	0.13
query27	0.04	0.04	0.05
query28	10.01	1.12	1.07
query29	12.56	3.26	3.22
query30	0.24	0.06	0.06
query31	2.87	0.38	0.37
query32	3.29	0.46	0.46
query33	2.97	2.98	3.04
query34	16.89	4.48	4.50
query35	4.55	4.48	4.48
query36	0.67	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.03	0.02
query40	0.18	0.13	0.13
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 106 s
Total hot run time: 32.64 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 3, 2024
Copy link
Contributor

github-actions bot commented Dec 3, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz xinyiZzz merged commit bb96c24 into apache:master Dec 4, 2024
24 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.x dev/3.0.x-conflict not-merge/2.1 reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants