Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[chore](ci) reduce insert rows to prevent MEM_LIMIT_EXCEEDED #45177

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

hello-stephen
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hello-stephen
Copy link
Contributor Author

run performance

@doris-robot
Copy link

TPC-H: Total hot run time: 40321 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2a28233ca119808ea74a9924f06fdd151c565849, data reload: false

------ Round 1 ----------------------------------
q1	17576	7524	7351	7351
q2	2046	177	165	165
q3	10608	1104	1201	1104
q4	10570	760	714	714
q5	7602	2821	2757	2757
q6	241	155	151	151
q7	1012	656	601	601
q8	9229	1905	1982	1905
q9	6796	6538	6517	6517
q10	7020	2318	2321	2318
q11	462	264	256	256
q12	431	224	221	221
q13	17764	3015	3007	3007
q14	256	219	209	209
q15	588	528	523	523
q16	647	600	585	585
q17	977	493	578	493
q18	7265	6780	6675	6675
q19	1324	1021	939	939
q20	487	183	181	181
q21	4060	3340	3368	3340
q22	375	309	318	309
Total cold run time: 107336 ms
Total hot run time: 40321 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7291	7299	7265	7265
q2	331	233	234	233
q3	2927	2919	3008	2919
q4	2111	1863	1908	1863
q5	5652	5676	5695	5676
q6	235	143	142	142
q7	2271	1826	1874	1826
q8	3399	3556	3560	3556
q9	9033	9099	9109	9099
q10	3660	3604	3572	3572
q11	610	516	497	497
q12	875	644	654	644
q13	11620	3286	3222	3222
q14	321	267	291	267
q15	586	539	529	529
q16	700	643	640	640
q17	1852	1661	1591	1591
q18	8228	7829	7549	7549
q19	1726	1642	1496	1496
q20	2098	1880	1880	1880
q21	5638	5586	5497	5497
q22	641	575	558	558
Total cold run time: 71805 ms
Total hot run time: 60521 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198287 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2a28233ca119808ea74a9924f06fdd151c565849, data reload: false

query1	1246	974	957	957
query2	6223	2190	2151	2151
query3	10926	4585	4375	4375
query4	67108	29148	23724	23724
query5	4973	462	457	457
query6	414	186	183	183
query7	5478	305	291	291
query8	340	221	224	221
query9	8665	2649	2632	2632
query10	461	241	232	232
query11	17057	15298	15968	15298
query12	148	100	108	100
query13	1456	427	424	424
query14	10027	7988	7547	7547
query15	217	194	193	193
query16	7666	485	474	474
query17	1034	564	566	564
query18	1476	299	293	293
query19	190	151	152	151
query20	117	108	110	108
query21	214	105	100	100
query22	4777	4471	4377	4377
query23	34899	34697	34536	34536
query24	5592	2501	2418	2418
query25	488	408	379	379
query26	635	148	148	148
query27	1954	285	285	285
query28	4600	2508	2464	2464
query29	700	437	439	437
query30	211	158	154	154
query31	1037	861	867	861
query32	68	64	59	59
query33	429	293	310	293
query34	947	523	537	523
query35	898	783	760	760
query36	1122	1001	989	989
query37	133	75	78	75
query38	4478	4410	4369	4369
query39	1536	1530	1489	1489
query40	213	100	96	96
query41	43	48	43	43
query42	106	102	101	101
query43	537	533	519	519
query44	1200	857	837	837
query45	189	168	180	168
query46	1186	733	727	727
query47	2047	1948	1953	1948
query48	427	315	314	314
query49	714	419	401	401
query50	826	403	382	382
query51	7483	7250	7095	7095
query52	96	88	113	88
query53	251	179	175	175
query54	508	397	396	396
query55	79	74	79	74
query56	247	233	234	233
query57	1277	1119	1139	1119
query58	220	224	211	211
query59	3143	3093	3070	3070
query60	261	251	249	249
query61	110	101	109	101
query62	785	706	696	696
query63	212	190	204	190
query64	1376	660	678	660
query65	3299	3184	3220	3184
query66	625	297	303	297
query67	16078	15667	15772	15667
query68	4297	545	562	545
query69	425	261	252	252
query70	1179	1150	1160	1150
query71	349	257	253	253
query72	6363	4142	4044	4044
query73	769	352	348	348
query74	10344	9093	9172	9093
query75	3420	2671	2651	2651
query76	1952	1096	1130	1096
query77	478	269	269	269
query78	10517	9468	9536	9468
query79	1796	595	619	595
query80	1427	436	438	436
query81	531	230	234	230
query82	1255	116	126	116
query83	176	137	141	137
query84	280	68	70	68
query85	1044	300	297	297
query86	412	298	303	298
query87	4778	4513	4547	4513
query88	3707	2173	2139	2139
query89	426	293	296	293
query90	1999	186	185	185
query91	141	105	107	105
query92	62	48	49	48
query93	2033	538	542	538
query94	821	299	296	296
query95	340	247	247	247
query96	639	274	291	274
query97	2888	2689	2692	2689
query98	214	195	200	195
query99	1625	1287	1325	1287
Total cold run time: 320165 ms
Total hot run time: 198287 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.9 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2a28233ca119808ea74a9924f06fdd151c565849, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.02
query3	0.24	0.07	0.07
query4	1.62	0.10	0.11
query5	0.41	0.42	0.43
query6	1.17	0.66	0.67
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.50	0.52
query10	0.55	0.56	0.59
query11	0.15	0.11	0.11
query12	0.13	0.11	0.11
query13	0.61	0.60	0.61
query14	2.81	2.82	2.73
query15	0.89	0.84	0.82
query16	0.42	0.40	0.38
query17	1.08	0.99	1.05
query18	0.22	0.21	0.20
query19	1.98	1.75	2.01
query20	0.01	0.01	0.01
query21	15.39	0.59	0.58
query22	2.47	2.74	2.01
query23	16.97	0.84	0.76
query24	3.13	1.15	1.60
query25	0.20	0.18	0.17
query26	0.47	0.13	0.14
query27	0.04	0.04	0.05
query28	10.10	1.12	1.08
query29	12.59	3.23	3.17
query30	0.25	0.06	0.06
query31	2.86	0.39	0.37
query32	3.28	0.47	0.47
query33	3.01	3.05	3.10
query34	16.78	4.48	4.47
query35	4.56	4.54	4.51
query36	0.66	0.48	0.51
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.26 s
Total hot run time: 32.9 s

@hello-stephen
Copy link
Contributor Author

run performance

@doris-robot
Copy link

TPC-H: Total hot run time: 40010 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 55817aeb382bae7fa4912c9bbd39655866c4285b, data reload: false

------ Round 1 ----------------------------------
q1	17586	7555	7228	7228
q2	2043	177	168	168
q3	10591	1055	1156	1055
q4	10787	779	765	765
q5	7627	2757	2691	2691
q6	237	147	147	147
q7	980	621	634	621
q8	9438	1881	1912	1881
q9	7122	6571	6591	6571
q10	7016	2303	2290	2290
q11	475	262	244	244
q12	430	215	215	215
q13	17930	3048	3069	3048
q14	257	223	221	221
q15	591	557	534	534
q16	650	603	562	562
q17	969	557	522	522
q18	7239	6714	6765	6714
q19	1333	1001	1000	1000
q20	456	175	175	175
q21	4015	3219	3047	3047
q22	373	311	320	311
Total cold run time: 108145 ms
Total hot run time: 40010 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7348	7461	7280	7280
q2	332	231	231	231
q3	3059	3091	3080	3080
q4	2128	1968	1942	1942
q5	5808	5845	5760	5760
q6	233	142	140	140
q7	2211	1827	1805	1805
q8	3415	3589	3485	3485
q9	9053	9044	9009	9009
q10	3597	3558	3557	3557
q11	606	519	516	516
q12	827	617	609	609
q13	12602	3257	3257	3257
q14	315	266	282	266
q15	570	527	525	525
q16	684	686	642	642
q17	1844	1621	1622	1621
q18	8330	7823	7627	7627
q19	1700	1582	1598	1582
q20	2115	1865	1874	1865
q21	5637	5528	5402	5402
q22	641	560	606	560
Total cold run time: 73055 ms
Total hot run time: 60761 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197245 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 55817aeb382bae7fa4912c9bbd39655866c4285b, data reload: false

query1	1267	973	929	929
query2	6223	2102	1998	1998
query3	10988	4518	4305	4305
query4	66524	29609	23876	23876
query5	4969	478	465	465
query6	411	204	197	197
query7	5550	292	290	290
query8	320	245	248	245
query9	8851	2737	2710	2710
query10	458	243	251	243
query11	17269	15229	15884	15229
query12	161	102	102	102
query13	1482	417	406	406
query14	10226	7240	7480	7240
query15	211	185	180	180
query16	7075	471	483	471
query17	1069	589	584	584
query18	1849	322	317	317
query19	213	167	158	158
query20	119	115	113	113
query21	212	109	107	107
query22	4649	4572	4424	4424
query23	35047	34508	34304	34304
query24	5361	2433	2513	2433
query25	489	382	381	381
query26	654	152	150	150
query27	1754	270	281	270
query28	4535	2443	2434	2434
query29	699	416	426	416
query30	208	176	151	151
query31	1023	842	846	842
query32	74	54	52	52
query33	476	283	307	283
query34	932	507	521	507
query35	897	769	782	769
query36	1068	990	971	971
query37	118	68	74	68
query38	4506	4366	4432	4366
query39	1551	1494	1459	1459
query40	206	100	96	96
query41	50	43	46	43
query42	115	107	103	103
query43	557	512	511	511
query44	1183	846	827	827
query45	189	170	177	170
query46	1190	742	714	714
query47	2074	2002	1900	1900
query48	434	307	325	307
query49	721	397	394	394
query50	834	398	391	391
query51	7416	7152	7300	7152
query52	112	86	87	86
query53	247	174	176	174
query54	507	414	395	395
query55	85	76	75	75
query56	249	241	232	232
query57	1243	1122	1138	1122
query58	217	206	206	206
query59	3274	2973	3070	2973
query60	277	249	247	247
query61	109	105	104	104
query62	785	664	684	664
query63	224	188	197	188
query64	1356	653	637	637
query65	3297	3227	3257	3227
query66	711	299	302	299
query67	16158	15604	15823	15604
query68	4097	555	543	543
query69	428	269	263	263
query70	1209	1128	1110	1110
query71	425	253	247	247
query72	6368	4020	4037	4020
query73	760	362	360	360
query74	10373	9030	9114	9030
query75	3411	2681	2673	2673
query76	1972	1060	1145	1060
query77	480	262	310	262
query78	10507	9402	9541	9402
query79	2170	605	604	604
query80	1418	410	419	410
query81	503	235	230	230
query82	1220	120	124	120
query83	174	145	138	138
query84	275	80	75	75
query85	991	295	298	295
query86	418	295	299	295
query87	4702	4635	4522	4522
query88	3622	2228	2177	2177
query89	423	310	298	298
query90	1950	188	192	188
query91	138	106	102	102
query92	59	51	58	51
query93	2985	536	533	533
query94	888	291	251	251
query95	353	245	243	243
query96	629	278	278	278
query97	2841	2711	2676	2676
query98	225	195	198	195
query99	1607	1311	1308	1308
Total cold run time: 320929 ms
Total hot run time: 197245 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.64 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 55817aeb382bae7fa4912c9bbd39655866c4285b, data reload: false

query1	0.04	0.03	0.02
query2	0.08	0.04	0.03
query3	0.24	0.08	0.07
query4	1.63	0.10	0.10
query5	0.42	0.41	0.43
query6	1.16	0.66	0.65
query7	0.02	0.01	0.02
query8	0.05	0.03	0.03
query9	0.58	0.49	0.50
query10	0.55	0.55	0.53
query11	0.13	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.59	0.60
query14	2.85	2.72	2.77
query15	0.91	0.83	0.83
query16	0.39	0.39	0.38
query17	1.08	1.07	1.06
query18	0.22	0.20	0.21
query19	1.95	2.04	1.84
query20	0.01	0.02	0.01
query21	15.36	0.60	0.59
query22	2.64	2.80	2.36
query23	17.18	0.87	0.83
query24	3.36	2.05	0.53
query25	0.22	0.10	0.04
query26	0.48	0.13	0.14
query27	0.04	0.04	0.04
query28	10.24	1.10	1.07
query29	12.53	3.28	3.27
query30	0.24	0.06	0.06
query31	2.87	0.39	0.37
query32	3.29	0.46	0.46
query33	2.95	3.04	3.04
query34	16.95	4.47	4.45
query35	4.55	4.58	4.44
query36	0.67	0.48	0.49
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 107.09 s
Total hot run time: 32.64 s

@hello-stephen
Copy link
Contributor Author

run vault_p0

@hello-stephen
Copy link
Contributor Author

run compile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants