Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](orc) check all the cases before build_search_argument #44615

Merged
merged 7 commits into from
Nov 29, 2024
Merged

[fix](orc) check all the cases before build_search_argument #44615

merged 7 commits into from
Nov 29, 2024

Conversation

suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Nov 26, 2024 •
edited by morningman
Loading

What problem does this PR solve?

In the old logic, the check_expr_can_push_down function does not check whether the orc::Literal are constructed successfully, but only checks during build_search_argument. However, if it is found that the orc::Literal fails to be constructed after builder->startNot, it will fail because the builder cannot end startNot.
Therefore, we advance the behavior of constructing orc::Literal to the check_expr_can_push_down function and save the result to the map, so that it will never fail in the build_search_argument phase.

Related PR: #43255

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223 suxiaogang223 marked this pull request as draft November 26, 2024 10:46
@suxiaogang223 suxiaogang223 marked this pull request as ready for review November 26, 2024 18:05
@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39864 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 55170503f5afd25455e0900a2575d77ca1976281, data reload: false

------ Round 1 ----------------------------------
q1	17582	7439	7287	7287
q2	2039	173	167	167
q3	10775	1144	1160	1144
q4	10533	740	675	675
q5	7607	2721	2697	2697
q6	235	148	146	146
q7	972	624	614	614
q8	9238	1832	1925	1832
q9	6566	6377	6386	6377
q10	7045	2287	2301	2287
q11	474	266	249	249
q12	418	213	213	213
q13	17795	3017	3014	3014
q14	242	210	212	210
q15	576	542	518	518
q16	666	585	594	585
q17	971	606	533	533
q18	7181	6870	6796	6796
q19	1349	1011	936	936
q20	470	179	179	179
q21	4141	3299	3091	3091
q22	388	320	314	314
Total cold run time: 107263 ms
Total hot run time: 39864 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7224	7264	7195	7195
q2	330	228	233	228
q3	2927	2826	2946	2826
q4	2052	1794	1795	1794
q5	5731	5689	5671	5671
q6	248	138	140	138
q7	2255	1865	1833	1833
q8	3415	3569	3501	3501
q9	8837	8890	8884	8884
q10	3606	3531	3543	3531
q11	604	513	491	491
q12	828	629	641	629
q13	11335	3255	3177	3177
q14	305	279	270	270
q15	596	524	524	524
q16	696	637	634	634
q17	1846	1613	1596	1596
q18	8279	7876	7508	7508
q19	1718	1665	1533	1533
q20	2120	1853	1895	1853
q21	5483	5415	5404	5404
q22	636	565	567	565
Total cold run time: 71071 ms
Total hot run time: 59785 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197432 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 55170503f5afd25455e0900a2575d77ca1976281, data reload: false

query1	1215	927	979	927
query2	6241	2109	2092	2092
query3	10803	4034	3925	3925
query4	67543	31932	23797	23797
query5	4972	473	465	465
query6	305	184	174	174
query7	4863	304	296	296
query8	288	234	232	232
query9	6250	2774	2765	2765
query10	422	273	237	237
query11	15571	15203	15882	15203
query12	167	103	107	103
query13	1040	432	420	420
query14	10529	7549	7322	7322
query15	218	183	190	183
query16	7168	458	480	458
query17	1054	585	596	585
query18	1872	316	319	316
query19	234	163	155	155
query20	130	115	118	115
query21	206	109	109	109
query22	4923	4673	4595	4595
query23	34747	34167	34453	34167
query24	5544	2529	2482	2482
query25	500	403	389	389
query26	680	145	149	145
query27	2255	287	299	287
query28	4244	2541	2505	2505
query29	691	444	428	428
query30	215	155	153	153
query31	1013	850	847	847
query32	71	59	59	59
query33	454	287	288	287
query34	939	521	535	521
query35	878	747	749	747
query36	1107	982	1001	982
query37	123	74	77	74
query38	4483	4390	4427	4390
query39	1530	1465	1489	1465
query40	210	107	105	105
query41	50	46	48	46
query42	115	140	103	103
query43	556	506	511	506
query44	1172	834	842	834
query45	183	172	167	167
query46	1159	686	710	686
query47	2063	1898	1949	1898
query48	409	313	313	313
query49	716	398	402	398
query50	882	396	399	396
query51	7397	7223	7097	7097
query52	100	88	88	88
query53	249	181	177	177
query54	519	410	390	390
query55	74	78	78	78
query56	241	280	234	234
query57	1288	1197	1164	1164
query58	212	215	211	211
query59	3223	3250	3076	3076
query60	272	247	245	245
query61	106	109	108	108
query62	809	675	671	671
query63	219	191	196	191
query64	1375	709	622	622
query65	3275	3216	3188	3188
query66	695	298	304	298
query67	16172	15708	15665	15665
query68	3632	588	561	561
query69	443	258	260	258
query70	1239	1146	1164	1146
query71	405	248	245	245
query72	6495	4087	4045	4045
query73	784	366	374	366
query74	9873	8960	9090	8960
query75	3415	2658	2666	2658
query76	1651	1052	1126	1052
query77	502	290	294	290
query78	10478	9516	9409	9409
query79	1599	604	611	604
query80	843	435	432	432
query81	517	231	223	223
query82	1281	117	123	117
query83	207	147	157	147
query84	283	72	69	69
query85	1012	296	285	285
query86	351	300	319	300
query87	4881	4597	4537	4537
query88	3779	2236	2231	2231
query89	434	297	297	297
query90	2063	190	187	187
query91	145	100	104	100
query92	65	50	53	50
query93	2220	531	542	531
query94	812	300	300	300
query95	345	256	252	252
query96	625	277	290	277
query97	2891	2687	2647	2647
query98	221	196	201	196
query99	2010	1303	1304	1303
Total cold run time: 314730 ms
Total hot run time: 197432 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.52 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 55170503f5afd25455e0900a2575d77ca1976281, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.04
query3	0.24	0.08	0.07
query4	1.60	0.10	0.10
query5	0.43	0.42	0.41
query6	1.15	0.65	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.51
query10	0.55	0.56	0.55
query11	0.14	0.10	0.10
query12	0.15	0.12	0.11
query13	0.60	0.62	0.60
query14	2.85	2.73	2.72
query15	0.91	0.83	0.83
query16	0.38	0.38	0.40
query17	1.05	1.07	1.04
query18	0.22	0.21	0.23
query19	1.97	1.88	2.01
query20	0.01	0.01	0.01
query21	15.37	0.61	0.57
query22	2.73	2.02	1.43
query23	16.96	1.12	0.81
query24	2.97	2.07	1.19
query25	0.21	0.23	0.05
query26	0.63	0.13	0.13
query27	0.04	0.04	0.04
query28	9.88	1.11	1.08
query29	12.54	3.25	3.26
query30	0.26	0.08	0.07
query31	2.83	0.38	0.38
query32	3.27	0.46	0.46
query33	2.98	3.08	3.06
query34	17.18	4.44	4.48
query35	4.48	4.55	4.48
query36	0.69	0.51	0.51
query37	0.10	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.47 s
Total hot run time: 32.52 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.35% (9977/26017)
Line Coverage: 29.45% (83525/283588)
Region Coverage: 28.59% (42962/150293)
Branch Coverage: 25.18% (21816/86630)
Coverage Report: http://coverage.selectdb-in.cc/coverage/55170503f5afd25455e0900a2575d77ca1976281_55170503f5afd25455e0900a2575d77ca1976281/report/index.html

@suxiaogang223
Copy link
Contributor Author

run buildall

@suxiaogang223 suxiaogang223 force-pushed the fix_orc_predicate_push_down branch from d2d6724 to f06e0b5 Compare November 27, 2024 02:26
@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39745 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f06e0b50cb695b0c9af5e3ef1b627b545846ddd4, data reload: false

------ Round 1 ----------------------------------
q1	17601	7455	7266	7266
q2	2045	169	186	169
q3	10564	1094	1198	1094
q4	10589	694	747	694
q5	7603	2661	2732	2661
q6	240	146	149	146
q7	1002	631	601	601
q8	9253	1818	1926	1818
q9	6519	6385	6402	6385
q10	7042	2296	2301	2296
q11	466	261	258	258
q12	397	220	217	217
q13	17790	3069	2977	2977
q14	243	206	206	206
q15	567	545	532	532
q16	671	607	584	584
q17	974	522	567	522
q18	7198	6634	6696	6634
q19	1328	1017	960	960
q20	481	179	181	179
q21	3971	3332	3234	3234
q22	390	330	312	312
Total cold run time: 106934 ms
Total hot run time: 39745 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7219	7204	7216	7204
q2	329	231	240	231
q3	2899	2819	2972	2819
q4	2044	1791	1817	1791
q5	5715	5699	5618	5618
q6	231	146	143	143
q7	2284	1819	1797	1797
q8	3413	3537	3508	3508
q9	8890	8960	8853	8853
q10	3580	3541	3572	3541
q11	603	498	505	498
q12	816	617	625	617
q13	12816	3224	3271	3224
q14	303	269	283	269
q15	576	548	527	527
q16	714	664	646	646
q17	1851	1621	1626	1621
q18	8341	7701	7515	7515
q19	1709	1562	1549	1549
q20	2135	1851	1888	1851
q21	5579	5487	5505	5487
q22	616	583	548	548
Total cold run time: 72663 ms
Total hot run time: 59857 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.36% (9980/26017)
Line Coverage: 29.47% (83562/283582)
Region Coverage: 28.59% (42969/150288)
Branch Coverage: 25.20% (21833/86626)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f06e0b50cb695b0c9af5e3ef1b627b545846ddd4_f06e0b50cb695b0c9af5e3ef1b627b545846ddd4/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 197189 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f06e0b50cb695b0c9af5e3ef1b627b545846ddd4, data reload: false

query1	1255	949	894	894
query2	6261	2072	2050	2050
query3	10858	4186	3976	3976
query4	67679	28198	23565	23565
query5	4946	455	447	447
query6	431	204	189	189
query7	5481	309	297	297
query8	311	234	226	226
query9	8600	2752	2751	2751
query10	437	261	245	245
query11	16913	15281	15878	15281
query12	152	102	110	102
query13	1451	421	420	420
query14	10883	7607	7555	7555
query15	205	183	187	183
query16	7119	495	514	495
query17	1025	584	566	566
query18	1833	309	297	297
query19	207	160	151	151
query20	114	114	111	111
query21	206	104	107	104
query22	4712	4316	4571	4316
query23	34995	34511	34447	34447
query24	5486	2613	2591	2591
query25	472	401	414	401
query26	653	163	156	156
query27	1871	296	283	283
query28	4241	2515	2457	2457
query29	680	419	417	417
query30	215	151	148	148
query31	1038	810	855	810
query32	64	54	59	54
query33	454	287	302	287
query34	924	533	526	526
query35	863	749	744	744
query36	1133	962	978	962
query37	128	74	81	74
query38	4551	4483	4430	4430
query39	1559	1480	1574	1480
query40	210	103	105	103
query41	49	48	47	47
query42	120	104	102	102
query43	568	520	512	512
query44	1222	823	803	803
query45	190	175	167	167
query46	1129	706	722	706
query47	2017	1895	1920	1895
query48	438	322	337	322
query49	725	395	414	395
query50	839	421	393	393
query51	7381	7180	7139	7139
query52	102	90	87	87
query53	259	178	180	178
query54	525	401	415	401
query55	80	75	77	75
query56	260	234	264	234
query57	1299	1174	1153	1153
query58	218	229	223	223
query59	3396	2995	2976	2976
query60	288	269	249	249
query61	116	105	122	105
query62	770	673	659	659
query63	218	194	194	194
query64	1385	658	678	658
query65	3255	3222	3235	3222
query66	775	298	294	294
query67	16061	15420	15740	15420
query68	3783	580	581	580
query69	430	258	258	258
query70	1232	1139	1060	1060
query71	356	263	247	247
query72	6374	4021	4120	4021
query73	775	372	366	366
query74	10191	8970	9068	8970
query75	3391	2709	2657	2657
query76	1791	1105	1059	1059
query77	512	317	275	275
query78	10487	9409	9348	9348
query79	1675	612	617	612
query80	1439	421	443	421
query81	531	231	230	230
query82	1320	121	118	118
query83	272	158	158	158
query84	282	76	71	71
query85	1032	290	296	290
query86	416	276	306	276
query87	4723	4567	4627	4567
query88	3536	2249	2212	2212
query89	417	303	306	303
query90	1985	190	188	188
query91	139	103	106	103
query92	63	57	53	53
query93	2065	541	563	541
query94	845	298	305	298
query95	356	264	256	256
query96	615	287	275	275
query97	2874	2696	2679	2679
query98	224	196	196	196
query99	1724	1318	1301	1301
Total cold run time: 320156 ms
Total hot run time: 197189 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.08 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f06e0b50cb695b0c9af5e3ef1b627b545846ddd4, data reload: false

query1	0.04	0.03	0.02
query2	0.06	0.04	0.03
query3	0.23	0.07	0.08
query4	1.60	0.10	0.10
query5	0.42	0.42	0.42
query6	1.14	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.59	0.52	0.51
query10	0.54	0.55	0.57
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.61	0.60
query14	2.69	2.82	2.74
query15	0.90	0.84	0.82
query16	0.39	0.39	0.39
query17	1.06	1.05	1.06
query18	0.22	0.22	0.21
query19	1.97	1.88	2.02
query20	0.02	0.01	0.01
query21	15.36	0.60	0.57
query22	2.85	2.20	1.96
query23	16.96	1.15	0.70
query24	3.02	1.45	1.15
query25	0.14	0.19	0.12
query26	0.49	0.14	0.14
query27	0.04	0.05	0.04
query28	10.07	1.11	1.07
query29	12.53	3.25	3.23
query30	0.25	0.06	0.06
query31	2.89	0.40	0.38
query32	3.28	0.49	0.48
query33	3.02	3.03	3.09
query34	16.91	4.50	4.58
query35	4.60	4.58	4.55
query36	0.67	0.49	0.48
query37	0.09	0.06	0.07
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.15	0.13	0.13
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.34 s
Total hot run time: 33.08 s

@suxiaogang223
Copy link
Contributor Author

run external

@suxiaogang223
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.34% (9976/26018)
Line Coverage: 29.45% (83573/283743)
Region Coverage: 28.57% (42967/150387)
Branch Coverage: 25.19% (21829/86648)
Coverage Report: http://coverage.selectdb-in.cc/coverage/91c17ee8d451792d45e5aedb71098b9a9441e707_91c17ee8d451792d45e5aedb71098b9a9441e707/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 39882 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 91c17ee8d451792d45e5aedb71098b9a9441e707, data reload: false

------ Round 1 ----------------------------------
q1	17570	7461	7243	7243
q2	2051	177	160	160
q3	10663	1095	1221	1095
q4	10565	727	748	727
q5	7884	2795	2720	2720
q6	241	160	156	156
q7	979	638	615	615
q8	9246	1863	1916	1863
q9	6603	6394	6448	6394
q10	7035	2323	2379	2323
q11	460	269	259	259
q12	423	218	213	213
q13	17759	3015	3046	3015
q14	255	211	217	211
q15	581	523	522	522
q16	646	604	592	592
q17	974	574	551	551
q18	7225	6581	6762	6581
q19	1310	983	1021	983
q20	477	178	178	178
q21	3981	3265	3171	3171
q22	378	313	310	310
Total cold run time: 107306 ms
Total hot run time: 39882 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7253	7307	7229	7229
q2	320	226	225	225
q3	2916	2777	2960	2777
q4	2063	1770	1864	1770
q5	5714	5701	5614	5614
q6	223	148	146	146
q7	2276	1834	1823	1823
q8	3403	3575	3527	3527
q9	8866	8920	8883	8883
q10	3623	3537	3561	3537
q11	596	506	515	506
q12	799	602	575	575
q13	11384	3296	3271	3271
q14	302	271	279	271
q15	591	523	527	523
q16	695	642	641	641
q17	1863	1647	1607	1607
q18	8263	7792	7514	7514
q19	1767	1678	1554	1554
q20	2147	1928	1897	1897
q21	5570	5292	5350	5292
q22	679	567	606	567
Total cold run time: 71313 ms
Total hot run time: 59749 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196808 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 91c17ee8d451792d45e5aedb71098b9a9441e707, data reload: false

query1	1250	944	939	939
query2	6228	2029	2029	2029
query3	10941	3958	4081	3958
query4	67593	28635	23589	23589
query5	4982	451	440	440
query6	409	207	182	182
query7	5501	307	293	293
query8	319	233	228	228
query9	8549	2730	2718	2718
query10	455	264	271	264
query11	17028	15212	16016	15212
query12	158	107	102	102
query13	1487	443	453	443
query14	10814	7136	6789	6789
query15	209	193	199	193
query16	7114	493	500	493
query17	1080	591	606	591
query18	1829	344	307	307
query19	202	163	149	149
query20	118	118	118	118
query21	209	107	113	107
query22	4571	4612	4565	4565
query23	35172	34694	34407	34407
query24	5633	2546	2539	2539
query25	505	392	397	392
query26	661	155	155	155
query27	1911	292	290	290
query28	4426	2502	2468	2468
query29	682	416	417	416
query30	220	176	149	149
query31	1024	834	867	834
query32	65	60	56	56
query33	404	287	297	287
query34	958	538	525	525
query35	900	772	762	762
query36	1081	973	975	973
query37	123	78	76	76
query38	4479	4485	4520	4485
query39	1526	1466	1485	1466
query40	206	109	98	98
query41	44	42	44	42
query42	113	98	98	98
query43	543	503	493	493
query44	1186	841	818	818
query45	187	169	171	169
query46	1159	730	713	713
query47	2067	1940	1964	1940
query48	424	338	321	321
query49	728	399	388	388
query50	849	398	416	398
query51	7373	7314	7155	7155
query52	98	89	87	87
query53	259	184	181	181
query54	526	415	388	388
query55	84	76	78	76
query56	246	236	265	236
query57	1318	1194	1178	1178
query58	223	222	219	219
query59	3237	2960	3030	2960
query60	270	250	240	240
query61	112	112	112	112
query62	772	656	677	656
query63	212	193	187	187
query64	1454	663	645	645
query65	3270	3219	3221	3219
query66	690	303	304	303
query67	16108	15636	15527	15527
query68	3733	571	579	571
query69	427	280	256	256
query70	1186	1094	1138	1094
query71	370	254	255	254
query72	6442	4110	3986	3986
query73	771	360	369	360
query74	10212	8934	8961	8934
query75	3409	2682	2650	2650
query76	1778	1109	1114	1109
query77	522	296	285	285
query78	10501	9550	9458	9458
query79	1840	599	608	599
query80	1250	423	476	423
query81	529	242	227	227
query82	460	118	115	115
query83	182	150	147	147
query84	290	73	68	68
query85	1005	308	305	305
query86	384	309	315	309
query87	4698	4476	4512	4476
query88	3570	2229	2182	2182
query89	421	312	295	295
query90	2034	192	191	191
query91	147	106	102	102
query92	70	49	52	49
query93	2546	549	548	548
query94	901	301	268	268
query95	354	251	249	249
query96	631	279	282	279
query97	2867	2685	2671	2671
query98	212	197	195	195
query99	2048	1320	1317	1317
Total cold run time: 320364 ms
Total hot run time: 196808 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.13 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 91c17ee8d451792d45e5aedb71098b9a9441e707, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.23	0.08	0.06
query4	1.62	0.10	0.10
query5	0.45	0.41	0.41
query6	1.13	0.66	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.52	0.49
query10	0.56	0.54	0.56
query11	0.14	0.11	0.11
query12	0.13	0.12	0.11
query13	0.60	0.59	0.61
query14	2.70	2.75	2.69
query15	0.90	0.83	0.82
query16	0.37	0.38	0.39
query17	1.06	1.06	1.05
query18	0.22	0.21	0.22
query19	1.95	1.87	1.91
query20	0.02	0.00	0.02
query21	15.42	0.62	0.60
query22	2.81	2.46	2.16
query23	17.13	0.86	0.86
query24	2.88	1.09	1.96
query25	0.20	0.06	0.08
query26	0.60	0.14	0.14
query27	0.04	0.04	0.04
query28	9.97	1.12	1.09
query29	12.59	3.28	3.20
query30	0.24	0.06	0.06
query31	2.88	0.39	0.38
query32	3.27	0.47	0.47
query33	3.01	3.08	3.04
query34	17.10	4.48	4.50
query35	4.54	4.54	4.49
query36	0.68	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.03	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.65 s
Total hot run time: 33.13 s

@suxiaogang223 @morningman
fix wrong logic
3df89ac 
Co-authored-by: Mingyu Chen (Rayner) <[email protected]>
@doris-robot
Copy link

TPC-H: Total hot run time: 39805 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e9abde09605a333e5576bc0154c75cb87751fc10, data reload: false

------ Round 1 ----------------------------------
q1	17601	7452	7236	7236
q2	2043	186	179	179
q3	10539	1132	1139	1132
q4	10229	753	678	678
q5	7610	2703	2716	2703
q6	235	146	145	145
q7	973	627	606	606
q8	9252	1820	1933	1820
q9	6642	6442	6466	6442
q10	6940	2309	2331	2309
q11	460	259	265	259
q12	414	219	212	212
q13	17765	2999	2999	2999
q14	248	210	213	210
q15	576	545	528	528
q16	626	579	585	579
q17	970	596	539	539
q18	7505	6627	6527	6527
q19	1356	1092	1003	1003
q20	486	195	181	181
q21	4055	3206	3195	3195
q22	400	324	323	323
Total cold run time: 106925 ms
Total hot run time: 39805 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7232	7276	7212	7212
q2	331	233	236	233
q3	2929	2730	3028	2730
q4	2168	1794	1820	1794
q5	5593	5700	5688	5688
q6	225	137	136	136
q7	2200	1784	1856	1784
q8	3379	3537	3496	3496
q9	8955	9080	8966	8966
q10	3564	3611	3505	3505
q11	595	532	497	497
q12	834	604	594	594
q13	13463	3185	3200	3185
q14	291	273	260	260
q15	582	529	513	513
q16	716	633	632	632
q17	1778	1573	1548	1548
q18	7814	7496	7299	7299
q19	1639	1526	1513	1513
q20	2040	1798	1786	1786
q21	5468	5217	5257	5217
q22	636	556	560	556
Total cold run time: 72432 ms
Total hot run time: 59144 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.33% (9978/26030)
Line Coverage: 29.42% (83554/284046)
Region Coverage: 28.53% (42984/150646)
Branch Coverage: 25.16% (21835/86790)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e9abde09605a333e5576bc0154c75cb87751fc10_e9abde09605a333e5576bc0154c75cb87751fc10/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 190650 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e9abde09605a333e5576bc0154c75cb87751fc10, data reload: false

query1	980	380	377	377
query2	6525	2123	2091	2091
query3	6712	213	218	213
query4	34025	23557	23535	23535
query5	4389	466	454	454
query6	279	189	175	175
query7	4615	311	308	308
query8	295	230	228	228
query9	9576	2732	2723	2723
query10	472	260	251	251
query11	18206	15235	15225	15225
query12	151	98	108	98
query13	1674	431	424	424
query14	10243	6503	6929	6503
query15	269	178	187	178
query16	8125	459	469	459
query17	1616	577	558	558
query18	2130	299	304	299
query19	360	148	189	148
query20	118	110	111	110
query21	211	98	99	98
query22	4592	4160	4206	4160
query23	34998	34214	35983	34214
query24	11456	2413	2452	2413
query25	524	383	374	374
query26	1117	147	150	147
query27	2889	280	277	277
query28	8034	2430	2447	2430
query29	646	395	397	395
query30	297	147	149	147
query31	1032	803	804	803
query32	91	55	56	55
query33	775	305	282	282
query34	986	533	536	533
query35	916	745	750	745
query36	1110	969	960	960
query37	201	72	70	70
query38	4337	4261	4305	4261
query39	1468	1442	1426	1426
query40	281	99	101	99
query41	48	43	42	42
query42	105	96	94	94
query43	516	490	492	490
query44	1203	790	786	786
query45	192	171	162	162
query46	1173	718	709	709
query47	1941	1868	1827	1827
query48	400	324	307	307
query49	1276	382	386	382
query50	803	391	369	369
query51	7135	6968	7142	6968
query52	101	87	85	85
query53	254	179	184	179
query54	1131	399	399	399
query55	79	81	79	79
query56	257	238	237	237
query57	1236	1141	1135	1135
query58	225	204	214	204
query59	3211	3031	2994	2994
query60	259	240	233	233
query61	106	105	101	101
query62	856	681	711	681
query63	211	182	184	182
query64	4876	655	607	607
query65	3286	3201	3248	3201
query66	1279	311	312	311
query67	16069	15798	15618	15618
query68	4546	559	589	559
query69	410	246	258	246
query70	1195	1091	1161	1091
query71	346	245	254	245
query72	6076	4061	4045	4045
query73	772	367	360	360
query74	10548	8954	9050	8954
query75	3429	2649	2645	2645
query76	2718	1083	1152	1083
query77	523	268	286	268
query78	10312	9360	9385	9360
query79	2224	599	606	599
query80	1052	418	422	418
query81	523	237	235	235
query82	634	117	124	117
query83	254	142	146	142
query84	240	68	67	67
query85	1880	294	298	294
query86	512	304	274	274
query87	4831	4690	4678	4678
query88	4101	2236	2190	2190
query89	405	301	301	301
query90	2184	190	194	190
query91	138	109	98	98
query92	66	52	51	51
query93	2329	529	542	529
query94	1140	295	289	289
query95	362	243	244	243
query96	640	278	281	278
query97	2888	2648	2706	2648
query98	212	199	203	199
query99	1594	1328	1326	1326
Total cold run time: 304033 ms
Total hot run time: 190650 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.12 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e9abde09605a333e5576bc0154c75cb87751fc10, data reload: false

query1	0.03	0.02	0.03
query2	0.06	0.03	0.04
query3	0.24	0.08	0.07
query4	1.60	0.10	0.10
query5	0.42	0.41	0.41
query6	1.17	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.58	0.50	0.48
query10	0.56	0.56	0.55
query11	0.14	0.11	0.11
query12	0.13	0.12	0.12
query13	0.62	0.62	0.59
query14	2.74	2.76	2.82
query15	0.90	0.83	0.83
query16	0.40	0.38	0.39
query17	1.01	1.06	1.05
query18	0.22	0.21	0.22
query19	1.97	1.84	2.04
query20	0.01	0.01	0.00
query21	15.36	0.59	0.59
query22	2.29	2.80	2.22
query23	16.98	1.06	0.83
query24	3.22	1.80	0.97
query25	0.25	0.12	0.13
query26	0.43	0.14	0.14
query27	0.05	0.04	0.04
query28	9.90	1.08	1.06
query29	12.52	3.24	3.24
query30	0.25	0.07	0.06
query31	2.87	0.38	0.37
query32	3.30	0.46	0.46
query33	2.96	3.07	3.22
query34	17.02	4.44	4.44
query35	4.59	4.47	4.49
query36	0.67	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.13
query41	0.07	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.99 s
Total hot run time: 33.12 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

kaka11chen
kaka11chen approved these changes Nov 29, 2024
View reviewed changes
Copy link
Contributor

@kaka11chen kaka11chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 08b187b into apache:master Nov 29, 2024
22 of 25 checks passed
morningman pushed a commit to morningman/doris that referenced this pull request Nov 29, 2024
@suxiaogang223 @morningman
[fix](orc) check all the cases before build_search_argument (apache#4…
0174028 
…4615)

In the old logic, the `check_expr_can_push_down` function does not check
whether the `orc::Literal` are constructed successfully, but only checks
during `build_search_argument`. However, if it is found that the
`orc::Literal` fails to be constructed after `builder->startNot`, it
will fail because the builder cannot end `startNot`.
Therefore, we advance the behavior of constructing `orc::Literal` to the
`check_expr_can_push_down` function and save the result to the map, so
that it will never fail in the `build_search_argument` phase.

Related PR: apache#43255
This was referenced Nov 29, 2024
Merged
Merged
morningman added a commit that referenced this pull request Nov 30, 2024
@morningman @suxiaogang223
[fix](orc) check all the cases before build_search_argument (#44615) (#…
88926d2 
…44802)

cherry-pick #44615

Co-authored-by: Socrates <[email protected]>
morningman added a commit that referenced this pull request Nov 30, 2024
@morningman @suxiaogang223
[fix](orc) check all the cases before build_search_argument (#44615) (#…
4b15b1f 
…44801)

cherry-pick #44615

Co-authored-by: Socrates <[email protected]>
morningman added a commit that referenced this pull request Dec 6, 2024
@morningman
Revert "[fix](orc) check all the cases before build_search_argument (#…
0714a3b 
…44615) …"

This reverts commit 4b15b1f.
morningman added a commit to morningman/doris that referenced this pull request Dec 6, 2024
@morningman
Revert "[fix](orc) check all the cases before build_search_argument (a…
c4c7218 
…pache#44615) (apache#44801)"

This reverts commit 4b15b1f.
@morningman morningman mentioned this pull request Dec 6, 2024
Merged
yiguolei pushed a commit that referenced this pull request Dec 6, 2024
@morningman
Revert #43255 & #44615 (#45096)
d4a6fd1 
Revert "branch-2.1: [enhance](orc) Optimize ORC Predicate Pushdown for
OR-connected Predicate #43255 (#44438)"
Revert "[fix](orc) check all the cases before build_search_argument
(#44615) (#44801)"
@suxiaogang223 suxiaogang223 deleted the fix_orc_predicate_push_down branch December 12, 2024 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@kaka11chen kaka11chen kaka11chen approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@suxiaogang223 @doris-robot @kaka11chen @morningman