Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feat](nereids)set runtime filter wait time according to table row count and table type #42640 branch-3.0 #44031

Merged
merged 5 commits into from
Nov 29, 2024
Merged

[feat](nereids)set runtime filter wait time according to table row count and table type #42640 branch-3.0 #44031

merged 5 commits into from
Nov 29, 2024

Conversation

englefly
Copy link
Contributor

What problem does this PR solve?

pick #42640

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

englefly and others added 2 commits November 15, 2024 16:35
@englefly
[feat](nereids)set runtime filter wait time according to table row co…
a72038b 
…unt and table type (apache#42640)

adjust compute rf wait time according to max table row count, if wait
time is not default value
- olap:
  -- row < 1G: 1 sec
  -- 1G <= row < 10G: 5 sec
  -- 10G < row: 20 sec
- external:
  -- row < 1G: 5 sec
  -- 1G <= row < 10G: 10 sec
  -- 10G < row: 50 sec

Issue Number: close #xxx

<!--Describe your changes.-->
@englefly
fmt
f9fa5eb
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring
Copy link
Contributor

run buildall

@englefly
Copy link
Contributor Author

run buildall

@englefly
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40475 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit be6883824c4ffee26c628f76ee6fdcd281833a4e, data reload: false

------ Round 1 ----------------------------------
q1	17581	7365	7315	7315
q2	2043	183	161	161
q3	10673	1069	1187	1069
q4	10558	772	793	772
q5	7733	2825	2791	2791
q6	235	148	146	146
q7	965	617	612	612
q8	9580	1921	1962	1921
q9	8387	6367	6431	6367
q10	6974	2242	2308	2242
q11	460	262	266	262
q12	402	220	215	215
q13	17788	2985	2982	2982
q14	245	211	205	205
q15	571	521	528	521
q16	656	594	607	594
q17	960	573	575	573
q18	7124	6528	6539	6528
q19	1372	1000	1037	1000
q20	469	201	193	193
q21	3956	3174	3032	3032
q22	1080	988	974	974
Total cold run time: 109812 ms
Total hot run time: 40475 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7255	7206	7225	7206
q2	326	224	229	224
q3	2897	2912	2863	2863
q4	2038	1769	1774	1769
q5	5660	5670	5736	5670
q6	217	141	140	140
q7	2170	1772	1736	1736
q8	3269	3537	3443	3443
q9	8833	8906	8882	8882
q10	3571	3491	3491	3491
q11	603	507	514	507
q12	796	625	635	625
q13	16554	3116	3138	3116
q14	307	265	263	263
q15	580	532	503	503
q16	716	680	665	665
q17	1841	1597	1582	1582
q18	8083	7722	7637	7637
q19	3955	1615	1518	1518
q20	2096	1884	1815	1815
q21	5380	5225	5220	5220
q22	1118	1029	1035	1029
Total cold run time: 78265 ms
Total hot run time: 59904 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194991 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be6883824c4ffee26c628f76ee6fdcd281833a4e, data reload: false

query1	1245	902	902	902
query2	6248	2144	2068	2068
query3	10764	3742	3701	3701
query4	68104	28756	23460	23460
query5	5478	442	423	423
query6	434	175	162	162
query7	6100	308	300	300
query8	315	221	224	221
query9	9441	2657	2653	2653
query10	519	260	255	255
query11	18107	15323	15770	15323
query12	152	102	103	102
query13	1586	431	428	428
query14	10630	6463	7065	6463
query15	213	174	182	174
query16	7229	534	484	484
query17	1122	577	578	577
query18	1905	328	311	311
query19	222	177	170	170
query20	116	113	115	113
query21	216	101	101	101
query22	4667	4546	4478	4478
query23	34580	33958	34118	33958
query24	5986	2857	2972	2857
query25	525	375	381	375
query26	662	156	161	156
query27	1880	287	299	287
query28	4326	2571	2550	2550
query29	649	442	448	442
query30	230	165	174	165
query31	1000	808	823	808
query32	62	54	50	50
query33	415	268	276	268
query34	915	505	512	505
query35	838	726	730	726
query36	1074	953	932	932
query37	120	70	71	70
query38	4135	3977	3893	3893
query39	1490	1448	1459	1448
query40	198	92	96	92
query41	48	44	45	44
query42	108	99	94	94
query43	540	488	503	488
query44	1150	787	795	787
query45	194	179	181	179
query46	1152	751	722	722
query47	1952	1924	1922	1922
query48	480	361	372	361
query49	715	394	367	367
query50	828	408	418	408
query51	7527	7249	7150	7150
query52	101	83	83	83
query53	255	180	182	180
query54	554	441	442	441
query55	74	72	76	72
query56	238	223	233	223
query57	1166	1072	1076	1072
query58	207	196	199	196
query59	3247	3054	2923	2923
query60	263	239	242	239
query61	99	98	102	98
query62	755	661	649	649
query63	207	184	186	184
query64	1472	647	692	647
query65	3243	3181	3160	3160
query66	735	312	311	311
query67	15734	15250	15334	15250
query68	4546	559	556	556
query69	426	268	264	264
query70	1099	1122	1115	1115
query71	357	241	250	241
query72	6632	4106	4120	4106
query73	758	334	333	333
query74	9227	8910	8988	8910
query75	3307	2616	2629	2616
query76	1907	1104	1065	1065
query77	498	287	286	286
query78	10720	9622	9484	9484
query79	8407	591	594	591
query80	2233	428	422	422
query81	577	241	238	238
query82	1341	121	115	115
query83	247	132	134	132
query84	283	77	75	75
query85	1736	300	285	285
query86	464	290	281	281
query87	4439	4215	4207	4207
query88	5732	2371	2362	2362
query89	560	283	285	283
query90	2101	185	182	182
query91	181	140	141	140
query92	63	48	48	48
query93	6601	549	554	549
query94	927	289	288	288
query95	345	250	250	250
query96	612	280	279	279
query97	3328	3110	3122	3110
query98	222	196	193	193
query99	1699	1284	1292	1284
Total cold run time: 338517 ms
Total hot run time: 194991 ms

@englefly
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40431 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d26f7aaeea4e9c872e3413adc0cf473f8d541621, data reload: false

------ Round 1 ----------------------------------
q1	17570	7314	7252	7252
q2	2054	180	157	157
q3	10707	1204	1149	1149
q4	10554	775	705	705
q5	7740	2759	2777	2759
q6	235	144	151	144
q7	943	604	594	594
q8	9578	1922	1945	1922
q9	8036	6319	6396	6319
q10	6991	2345	2294	2294
q11	456	266	263	263
q12	395	216	205	205
q13	17758	2977	2983	2977
q14	229	212	205	205
q15	561	514	525	514
q16	674	588	603	588
q17	955	527	593	527
q18	7105	6541	6541	6541
q19	3257	1018	1085	1018
q20	470	195	195	195
q21	3890	3132	3185	3132
q22	1079	971	975	971
Total cold run time: 111237 ms
Total hot run time: 40431 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7434	7217	7202	7202
q2	309	228	233	228
q3	2977	2869	2853	2853
q4	2066	1759	1801	1759
q5	5615	5713	5840	5713
q6	214	139	142	139
q7	2153	1724	1713	1713
q8	3321	3398	3434	3398
q9	8911	8866	8782	8782
q10	3564	3515	3499	3499
q11	592	496	498	496
q12	825	617	604	604
q13	16391	3167	3155	3155
q14	303	279	261	261
q15	560	525	517	517
q16	692	674	677	674
q17	1833	1560	1600	1560
q18	8128	7803	7364	7364
q19	1694	1459	1545	1459
q20	2066	1839	1829	1829
q21	5457	5333	5435	5333
q22	1083	1005	1017	1005
Total cold run time: 76188 ms
Total hot run time: 59543 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194867 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d26f7aaeea4e9c872e3413adc0cf473f8d541621, data reload: false

query1	1205	913	891	891
query2	6244	2014	2042	2014
query3	10758	3815	3571	3571
query4	67184	30132	23458	23458
query5	5151	432	420	420
query6	417	170	158	158
query7	5617	316	309	309
query8	307	214	223	214
query9	9487	2661	2651	2651
query10	476	268	243	243
query11	17559	15334	15816	15334
query12	154	97	108	97
query13	1517	446	436	436
query14	11017	6870	7087	6870
query15	214	177	179	177
query16	7508	492	488	488
query17	1041	587	571	571
query18	1882	332	309	309
query19	197	154	146	146
query20	116	106	111	106
query21	209	111	99	99
query22	4663	4568	4480	4480
query23	34592	33901	34329	33901
query24	6296	2850	2864	2850
query25	503	377	384	377
query26	668	162	163	162
query27	1762	297	311	297
query28	4706	2582	2532	2532
query29	676	469	418	418
query30	229	155	160	155
query31	985	814	859	814
query32	64	54	53	53
query33	396	275	260	260
query34	901	488	498	488
query35	831	733	717	717
query36	1079	937	955	937
query37	122	69	75	69
query38	4122	4039	4090	4039
query39	1510	1444	1470	1444
query40	195	97	95	95
query41	50	45	46	45
query42	110	98	96	96
query43	522	494	494	494
query44	1154	798	824	798
query45	188	165	165	165
query46	1127	733	707	707
query47	1989	1882	1898	1882
query48	482	381	388	381
query49	740	402	384	384
query50	821	436	417	417
query51	7428	7187	7067	7067
query52	96	89	85	85
query53	268	181	181	181
query54	560	462	452	452
query55	75	74	73	73
query56	245	238	243	238
query57	1172	1106	1086	1086
query58	214	205	200	200
query59	3131	3023	3013	3013
query60	271	262	254	254
query61	122	135	96	96
query62	765	656	656	656
query63	209	182	184	182
query64	1409	634	629	629
query65	3235	3141	3154	3141
query66	705	282	304	282
query67	15899	15285	15256	15256
query68	4520	556	549	549
query69	430	254	258	254
query70	1168	1096	1068	1068
query71	332	244	246	244
query72	6381	3904	3889	3889
query73	746	338	330	330
query74	10127	8861	8888	8861
query75	3220	2603	2631	2603
query76	1756	1115	1120	1115
query77	471	250	248	248
query78	10564	9766	9428	9428
query79	7950	583	585	583
query80	2359	411	430	411
query81	567	244	239	239
query82	1480	117	144	117
query83	284	135	130	130
query84	288	82	76	76
query85	1630	298	284	284
query86	486	305	307	305
query87	4491	4210	4275	4210
query88	5845	2420	2428	2420
query89	419	287	297	287
query90	1990	185	185	185
query91	171	141	140	140
query92	72	45	50	45
query93	6721	564	551	551
query94	906	287	275	275
query95	356	263	251	251
query96	637	281	288	281
query97	3302	3103	3138	3103
query98	220	194	194	194
query99	1825	1294	1281	1281
Total cold run time: 337261 ms
Total hot run time: 194867 ms

@dataroaring dataroaring merged commit ec09d89 into apache:branch-3.0 Nov 29, 2024
18 of 20 checks passed
@englefly englefly deleted the pick-42640-3.0 branch November 30, 2024 03:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@englefly @doris-robot @dataroaring