Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [Opt](SegmentIterator) clear and release iterators memory footprint in advance when EOF #44768 #44839

Merged
merged 1 commit into from
Dec 2, 2024

Conversation

github-actions[bot]
Copy link
Contributor

@github-actions github-actions bot commented Dec 2, 2024

Cherry-picked from #44768

…n advance when EOF (#44768)

Otherwise the memory will hold until BetaRowsetReader released
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Dec 2, 2024
@dataroaring dataroaring reopened this Dec 2, 2024
@doris-robot
Copy link

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40278 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e598e98a93190d6d444e3d58b1c0e93258a099a4, data reload: false

------ Round 1 ----------------------------------
q1	17673	7607	7289	7289
q2	2039	164	160	160
q3	10979	1075	1146	1075
q4	10557	780	690	690
q5	7734	2816	2822	2816
q6	231	145	144	144
q7	976	609	617	609
q8	9552	1900	1962	1900
q9	7142	6399	6358	6358
q10	6999	2297	2251	2251
q11	444	257	261	257
q12	393	217	215	215
q13	17771	2982	2941	2941
q14	247	211	202	202
q15	575	521	516	516
q16	668	617	607	607
q17	957	529	613	529
q18	7134	6512	6515	6512
q19	1470	1083	997	997
q20	482	200	198	198
q21	3886	3174	3072	3072
q22	1081	968	940	940
Total cold run time: 108990 ms
Total hot run time: 40278 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7318	7254	7249	7249
q2	324	228	224	224
q3	2841	2904	2893	2893
q4	2004	1811	1817	1811
q5	5644	5705	5675	5675
q6	215	141	144	141
q7	2172	1713	1735	1713
q8	3276	3531	3454	3454
q9	8686	8837	8823	8823
q10	3534	3510	3538	3510
q11	588	488	495	488
q12	785	589	601	589
q13	16419	3190	3118	3118
q14	303	281	263	263
q15	564	515	524	515
q16	704	671	664	664
q17	1834	1671	1587	1587
q18	8131	7773	7485	7485
q19	6456	1574	1550	1550
q20	2051	1854	1858	1854
q21	5243	5351	5173	5173
q22	1143	1037	1024	1024
Total cold run time: 80235 ms
Total hot run time: 59803 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195276 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e598e98a93190d6d444e3d58b1c0e93258a099a4, data reload: false

query1	1253	909	913	909
query2	6203	2106	1986	1986
query3	10877	3963	3853	3853
query4	67551	29365	23749	23749
query5	5555	443	457	443
query6	459	179	172	172
query7	6159	311	324	311
query8	322	219	221	219
query9	9453	2660	2629	2629
query10	522	271	249	249
query11	18095	15324	15828	15324
query12	155	102	99	99
query13	1565	453	422	422
query14	10959	7056	6513	6513
query15	214	179	194	179
query16	7439	474	490	474
query17	1254	576	589	576
query18	1879	317	319	317
query19	210	159	151	151
query20	114	110	116	110
query21	239	106	98	98
query22	4593	4602	4477	4477
query23	34427	33899	33987	33899
query24	5969	2911	2959	2911
query25	526	399	378	378
query26	690	166	162	162
query27	2095	307	299	299
query28	4062	2529	2511	2511
query29	679	460	434	434
query30	235	158	162	158
query31	1002	814	829	814
query32	62	55	55	55
query33	430	265	266	265
query34	886	512	513	512
query35	829	719	728	719
query36	1085	951	940	940
query37	115	69	72	69
query38	4038	3983	4089	3983
query39	1531	1494	1478	1478
query40	211	99	96	96
query41	48	46	46	46
query42	103	98	100	98
query43	530	483	489	483
query44	1182	812	823	812
query45	185	173	172	172
query46	1153	724	775	724
query47	1944	1897	1908	1897
query48	462	373	375	373
query49	756	408	419	408
query50	832	445	426	426
query51	7260	7212	7020	7020
query52	103	89	90	89
query53	257	191	185	185
query54	574	462	453	453
query55	79	74	77	74
query56	259	250	247	247
query57	1185	1116	1090	1090
query58	217	210	215	210
query59	3207	2863	2923	2863
query60	283	264	258	258
query61	128	122	125	122
query62	773	672	658	658
query63	209	188	191	188
query64	1740	737	634	634
query65	3235	3154	3161	3154
query66	725	297	292	292
query67	15980	15147	15402	15147
query68	4558	560	537	537
query69	438	256	264	256
query70	1168	1118	1096	1096
query71	423	253	248	248
query72	6483	4000	3918	3918
query73	742	343	337	337
query74	9308	9121	9014	9014
query75	3422	2670	2652	2652
query76	2452	1136	1093	1093
query77	492	256	266	256
query78	10854	9645	9428	9428
query79	12056	600	598	598
query80	2292	421	419	419
query81	556	248	237	237
query82	1466	116	109	109
query83	308	139	138	138
query84	290	98	80	80
query85	1800	304	292	292
query86	460	293	282	282
query87	4487	4301	4230	4230
query88	6172	2362	2373	2362
query89	555	292	288	288
query90	2210	181	181	181
query91	189	140	141	140
query92	61	47	49	47
query93	7616	549	544	544
query94	920	288	292	288
query95	348	255	252	252
query96	646	295	274	274
query97	3395	3104	3139	3104
query98	217	198	194	194
query99	1605	1316	1344	1316
Total cold run time: 345340 ms
Total hot run time: 195276 ms

@eldenmoon eldenmoon merged commit 9affc10 into branch-3.0 Dec 2, 2024
18 of 21 checks passed
@github-actions github-actions bot deleted the auto-pick-44768-branch-3.0 branch December 2, 2024 09:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants