Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat](Nereids) support set var in hint when parse sql (#41331) #42335

Merged
merged 3 commits into from
Oct 25, 2024

Conversation

LiBinfeng-01
Copy link
Collaborator

pick: #41331
set var hint need to be enable to use before analyze, so it need to be set when parsing sql
now it would set twice when parse and begin of analyze

Proposed changes

Issue Number: close #xxx

set var hint need to be enable to use before analyze, so it need to be
set when parsing sql
now it would set twice when parse and begin of analyze
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49041 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d28d6c58067ff31c34dfa40ec591c48bc3abfdf1, data reload: false

------ Round 1 ----------------------------------
q1	18131	4382	4369	4369
q2	2070	153	143	143
q3	10427	1902	1938	1902
q4	10322	1262	1330	1262
q5	8415	3970	3902	3902
q6	233	122	124	122
q7	2026	1626	1598	1598
q8	9271	2715	2701	2701
q9	10268	9728	9706	9706
q10	8613	3503	3510	3503
q11	423	256	247	247
q12	466	299	295	295
q13	18351	3967	4042	3967
q14	343	328	324	324
q15	512	469	465	465
q16	547	461	482	461
q17	1128	985	952	952
q18	7335	6815	6850	6815
q19	1698	1571	1516	1516
q20	523	309	298	298
q21	4400	4153	4098	4098
q22	505	395	406	395
Total cold run time: 116007 ms
Total hot run time: 49041 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4320	4320	4333	4320
q2	330	224	225	224
q3	4176	4176	4188	4176
q4	2769	2742	2744	2742
q5	7097	6983	7046	6983
q6	236	128	119	119
q7	3252	2850	2877	2850
q8	4396	4453	4444	4444
q9	13763	13621	13608	13608
q10	4229	4241	4262	4241
q11	747	684	694	684
q12	1017	862	871	862
q13	6794	3780	3754	3754
q14	457	431	436	431
q15	513	461	454	454
q16	651	579	596	579
q17	3822	3823	3792	3792
q18	8880	8883	8885	8883
q19	1720	1674	1679	1674
q20	2356	2105	2103	2103
q21	8537	8531	8427	8427
q22	1056	953	922	922
Total cold run time: 81118 ms
Total hot run time: 76272 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 211288 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d28d6c58067ff31c34dfa40ec591c48bc3abfdf1, data reload: false

query1	929	426	380	380
query2	6538	2329	2025	2025
query3	6943	202	198	198
query4	23477	21476	21405	21405
query5	19744	6523	6577	6523
query6	288	214	238	214
query7	4330	288	301	288
query8	259	287	224	224
query9	3050	2658	2560	2560
query10	471	304	294	294
query11	16096	14898	15052	14898
query12	124	75	74	74
query13	1026	467	427	427
query14	17314	13123	13260	13123
query15	385	221	230	221
query16	6477	297	260	260
query17	1761	931	899	899
query18	893	320	310	310
query19	207	143	146	143
query20	103	94	94	94
query21	198	99	97	97
query22	5281	5011	5058	5011
query23	34394	33569	33436	33436
query24	7849	6316	6312	6312
query25	541	437	427	427
query26	1272	160	156	156
query27	2417	289	286	286
query28	6134	2228	2200	2200
query29	2937	2761	2868	2761
query30	242	167	176	167
query31	969	734	716	716
query32	72	59	58	58
query33	463	264	268	264
query34	864	464	474	464
query35	1126	932	943	932
query36	1211	1069	1274	1069
query37	169	62	61	61
query38	3094	2905	2906	2905
query39	1373	1346	1323	1323
query40	312	96	91	91
query41	39	37	37	37
query42	82	85	87	85
query43	651	613	628	613
query44	1172	719	713	713
query45	253	228	229	228
query46	1231	967	951	951
query47	1842	1678	1749	1678
query48	494	425	401	401
query49	673	374	407	374
query50	867	630	593	593
query51	4780	4632	4689	4632
query52	95	85	78	78
query53	219	185	193	185
query54	2663	2502	2481	2481
query55	86	87	84	84
query56	232	212	202	202
query57	1213	1218	1140	1140
query58	229	206	213	206
query59	3549	3421	3366	3366
query60	210	213	198	198
query61	107	93	98	93
query62	849	503	428	428
query63	200	173	174	173
query64	3496	1594	1341	1341
query65	3639	3533	3518	3518
query66	765	411	411	411
query67	18464	16409	15345	15345
query68	9122	647	653	647
query69	496	270	269	269
query70	1552	1390	1357	1357
query71	399	319	318	318
query72	6841	4731	4847	4731
query73	764	314	312	312
query74	6325	5755	5814	5755
query75	4525	3691	3664	3664
query76	4690	1173	1142	1142
query77	607	249	265	249
query78	12603	11529	11644	11529
query79	8945	635	675	635
query80	1740	393	381	381
query81	498	243	235	235
query82	1640	99	96	96
query83	170	128	139	128
query84	256	70	71	70
query85	969	312	316	312
query86	339	294	296	294
query87	3250	3051	2997	2997
query88	4713	2314	2290	2290
query89	488	278	305	278
query90	1888	213	216	213
query91	155	126	128	126
query92	56	54	51	51
query93	7086	567	548	548
query94	798	210	211	210
query95	1988	1935	1964	1935
query96	649	324	318	318
query97	6488	6338	6258	6258
query98	231	209	203	203
query99	2972	923	887	887
Total cold run time: 322676 ms
Total hot run time: 211288 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.48 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d28d6c58067ff31c34dfa40ec591c48bc3abfdf1, data reload: false

query1	0.02	0.02	0.03
query2	0.07	0.03	0.02
query3	0.25	0.05	0.05
query4	1.79	0.07	0.07
query5	0.54	0.52	0.52
query6	1.23	0.62	0.63
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.52	0.49	0.47
query10	0.54	0.53	0.51
query11	0.13	0.08	0.08
query12	0.12	0.09	0.09
query13	0.62	0.61	0.61
query14	0.78	0.78	0.78
query15	0.79	0.76	0.76
query16	0.36	0.36	0.37
query17	1.03	1.01	1.04
query18	0.22	0.27	0.25
query19	1.99	1.88	1.87
query20	0.02	0.01	0.01
query21	15.45	0.55	0.56
query22	2.40	2.15	1.47
query23	17.26	1.09	1.07
query24	5.98	0.89	1.20
query25	0.37	0.12	0.04
query26	0.63	0.16	0.14
query27	0.04	0.04	0.04
query28	7.09	0.72	0.71
query29	12.60	2.27	2.05
query30	0.58	0.52	0.52
query31	2.82	0.37	0.37
query32	3.39	0.50	0.49
query33	3.06	3.09	3.13
query34	15.27	4.82	4.82
query35	4.86	4.85	4.83
query36	1.06	1.02	1.01
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.15
query41	0.08	0.01	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 104.33 s
Total hot run time: 30.48 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d28d6c58067ff31c34dfa40ec591c48bc3abfdf1 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.9 seconds inserted 10000000 Rows, about 456K ops/s

@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 48862 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6941e36cb6a19e1823876887ff43d0419ef60f46, data reload: false

------ Round 1 ----------------------------------
q1	18205	4335	4351	4335
q2	2059	149	143	143
q3	10296	1919	1922	1919
q4	10351	1263	1306	1263
q5	8679	3789	3903	3789
q6	233	120	119	119
q7	2015	1615	1596	1596
q8	9283	2718	2701	2701
q9	10180	9981	9752	9752
q10	8624	3495	3478	3478
q11	424	239	250	239
q12	469	292	297	292
q13	18347	3981	3991	3981
q14	356	321	325	321
q15	510	458	455	455
q16	552	445	473	445
q17	1131	961	908	908
q18	7167	6782	6797	6782
q19	1699	1580	1560	1560
q20	524	306	296	296
q21	4464	4127	4102	4102
q22	503	386	405	386
Total cold run time: 116071 ms
Total hot run time: 48862 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4342	4297	4321	4297
q2	318	228	226	226
q3	4237	4193	4145	4145
q4	2751	2773	2764	2764
q5	7139	7102	7127	7102
q6	241	121	123	121
q7	3290	2834	2824	2824
q8	4363	4469	4517	4469
q9	13683	13661	13977	13661
q10	4420	4422	4441	4422
q11	800	728	679	679
q12	1088	918	943	918
q13	7841	3843	3762	3762
q14	453	429	435	429
q15	502	462	470	462
q16	624	576	608	576
q17	3838	3904	3854	3854
q18	8823	8803	8799	8799
q19	1706	1633	1663	1633
q20	2371	2122	2143	2122
q21	8457	8427	8400	8400
q22	1028	994	940	940
Total cold run time: 82315 ms
Total hot run time: 76605 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 212011 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6941e36cb6a19e1823876887ff43d0419ef60f46, data reload: false

query1	942	400	409	400
query2	6535	2187	2226	2187
query3	6921	203	208	203
query4	23184	21380	21445	21380
query5	19727	6491	6512	6491
query6	304	214	243	214
query7	4489	296	299	296
query8	258	258	235	235
query9	3044	2648	2585	2585
query10	472	322	291	291
query11	15763	15576	15051	15051
query12	135	84	74	74
query13	1025	430	435	430
query14	18071	13462	13115	13115
query15	369	221	225	221
query16	6460	277	259	259
query17	1776	929	908	908
query18	886	315	302	302
query19	203	154	144	144
query20	101	104	97	97
query21	186	96	94	94
query22	5248	5093	5108	5093
query23	34227	33436	33339	33339
query24	6680	6314	6354	6314
query25	527	439	423	423
query26	1013	162	157	157
query27	2325	290	283	283
query28	6062	2259	2204	2204
query29	2967	2771	2707	2707
query30	241	166	164	164
query31	938	715	736	715
query32	69	67	59	59
query33	436	264	256	256
query34	865	473	467	467
query35	1121	937	912	912
query36	1398	1241	1317	1241
query37	92	60	62	60
query38	3135	3000	2970	2970
query39	1372	1313	1319	1313
query40	239	93	93	93
query41	40	39	38	38
query42	85	82	89	82
query43	657	570	625	570
query44	1254	718	721	718
query45	239	229	228	228
query46	1232	968	974	968
query47	1899	2015	2034	2015
query48	510	423	402	402
query49	616	371	388	371
query50	873	608	616	608
query51	4856	4658	4675	4658
query52	87	85	77	77
query53	230	190	190	190
query54	2664	2459	2484	2459
query55	86	83	83	83
query56	211	194	196	194
query57	1270	1162	1213	1162
query58	220	216	203	203
query59	3480	3240	3210	3210
query60	222	196	196	196
query61	94	91	91	91
query62	814	496	435	435
query63	198	171	186	171
query64	3360	1522	1457	1457
query65	3642	3536	3521	3521
query66	782	394	409	394
query67	15940	14865	17158	14865
query68	9103	634	629	629
query69	496	271	277	271
query70	1646	1484	1524	1484
query71	393	302	312	302
query72	7083	4804	4732	4732
query73	746	320	313	313
query74	6333	5834	5787	5787
query75	4516	3628	3673	3628
query76	4574	1159	1154	1154
query77	569	251	253	251
query78	12689	11756	12089	11756
query79	7243	659	639	639
query80	2685	394	385	385
query81	516	237	235	235
query82	1713	97	93	93
query83	185	128	132	128
query84	262	68	71	68
query85	1368	311	309	309
query86	363	294	291	291
query87	3241	3050	3037	3037
query88	4756	2266	2287	2266
query89	421	305	318	305
query90	1763	213	218	213
query91	166	122	129	122
query92	61	50	50	50
query93	5289	554	602	554
query94	899	205	210	205
query95	2080	1957	1965	1957
query96	636	316	313	313
query97	6459	6328	6387	6328
query98	218	222	210	210
query99	2810	1050	883	883
Total cold run time: 316649 ms
Total hot run time: 212011 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.17 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6941e36cb6a19e1823876887ff43d0419ef60f46, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.02	0.02
query3	0.24	0.04	0.04
query4	1.79	0.05	0.06
query5	0.54	0.53	0.52
query6	1.24	0.62	0.60
query7	0.01	0.01	0.01
query8	0.03	0.03	0.02
query9	0.53	0.48	0.49
query10	0.55	0.54	0.53
query11	0.12	0.08	0.09
query12	0.12	0.09	0.09
query13	0.62	0.62	0.62
query14	0.77	0.78	0.80
query15	0.78	0.75	0.76
query16	0.35	0.36	0.37
query17	0.98	1.00	1.00
query18	0.22	0.27	0.24
query19	1.93	1.82	1.86
query20	0.01	0.01	0.01
query21	15.48	0.55	0.54
query22	2.12	1.99	1.15
query23	16.70	0.96	0.84
query24	5.29	1.72	1.14
query25	0.37	0.12	0.05
query26	0.69	0.16	0.15
query27	0.05	0.05	0.03
query28	7.05	0.70	0.69
query29	12.82	2.33	2.29
query30	0.60	0.52	0.52
query31	2.80	0.38	0.36
query32	3.41	0.49	0.50
query33	3.10	3.05	3.04
query34	15.24	4.77	4.77
query35	4.86	4.81	4.84
query36	1.04	1.01	1.02
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.14
query41	0.06	0.01	0.01
query42	0.02	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 102.92 s
Total hot run time: 30.17 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 6941e36cb6a19e1823876887ff43d0419ef60f46 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

@LiBinfeng-01
Copy link
Collaborator Author

run p0

3 similar comments
@LiBinfeng-01
Copy link
Collaborator Author

run p0

@LiBinfeng-01
Copy link
Collaborator Author

run p0

@LiBinfeng-01
Copy link
Collaborator Author

run p0

@starocean999 starocean999 merged commit c8b1f75 into apache:branch-2.0 Oct 25, 2024
22 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants