Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](HMSExternalTable) refactor code #46466

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

[fix](HMSExternalTable) refactor code #46466

wants to merge 5 commits into from
+241 −66

Conversation

BePPPower
Copy link
Contributor

@BePPPower BePPPower commented Jan 6, 2025 •
edited
Loading

Modifications: Refactor Code in HMSExternalTable.java:

Introduced a HMSDlaTable interface with multiple abstract methods.
Modified the HMSExternalTable to use HMSDlaTable for various partition-related methods.
Added a new class HiveExternalTable that implements the HMSDlaTable interface.

These changes focus on code refactoring to improve maintainability and modularity without altering the existing logic.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@BePPPower
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32716 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0a3cd2502d25b0508440502d19c82e3489892601, data reload: false

------ Round 1 ----------------------------------
q1	17573	6264	6055	6055
q2	2045	316	187	187
q3	10394	1243	768	768
q4	10247	862	440	440
q5	8194	2216	1959	1959
q6	206	183	151	151
q7	894	757	602	602
q8	9233	1358	1194	1194
q9	5325	4911	4903	4903
q10	6809	2322	1901	1901
q11	498	295	290	290
q12	361	370	228	228
q13	17766	3716	3019	3019
q14	237	233	218	218
q15	578	509	494	494
q16	651	633	587	587
q17	568	840	326	326
q18	6911	6367	6524	6367
q19	1801	960	566	566
q20	313	323	190	190
q21	2794	2148	1961	1961
q22	371	339	310	310
Total cold run time: 103769 ms
Total hot run time: 32716 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6260	6352	6259	6259
q2	238	337	239	239
q3	2303	2622	2361	2361
q4	1403	1879	1366	1366
q5	4338	4791	4838	4791
q6	184	184	141	141
q7	2094	1989	1804	1804
q8	2659	2832	2684	2684
q9	7316	7234	7286	7234
q10	3122	3358	2894	2894
q11	580	507	493	493
q12	652	731	640	640
q13	3452	3909	3281	3281
q14	277	294	294	294
q15	591	530	510	510
q16	647	685	662	662
q17	1248	1800	1265	1265
q18	7745	7509	7217	7217
q19	899	1170	1082	1082
q20	1998	2022	1982	1982
q21	5843	5435	4992	4992
q22	653	620	606	606
Total cold run time: 54502 ms
Total hot run time: 52797 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197715 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0a3cd2502d25b0508440502d19c82e3489892601, data reload: false

query1	1320	979	908	908
query2	6491	2395	2364	2364
query3	11132	4786	4689	4689
query4	33088	23693	24212	23693
query5	4359	627	456	456
query6	298	207	193	193
query7	3997	491	309	309
query8	302	246	219	219
query9	9222	2655	2635	2635
query10	446	307	243	243
query11	18074	15345	15219	15219
query12	151	104	99	99
query13	1573	536	428	428
query14	10539	6923	7445	6923
query15	240	201	194	194
query16	7793	626	482	482
query17	1548	747	618	618
query18	2006	405	313	313
query19	245	196	160	160
query20	125	121	124	121
query21	204	118	108	108
query22	4756	4855	4575	4575
query23	34870	33783	34166	33783
query24	6383	2296	2407	2296
query25	481	455	388	388
query26	706	280	149	149
query27	1978	497	324	324
query28	5117	2483	2461	2461
query29	566	563	423	423
query30	211	183	154	154
query31	1000	930	825	825
query32	71	62	55	55
query33	486	378	302	302
query34	758	864	509	509
query35	840	822	760	760
query36	1031	1072	996	996
query37	124	105	82	82
query38	4151	4192	4283	4192
query39	1529	1488	1476	1476
query40	208	133	103	103
query41	45	43	44	43
query42	121	103	103	103
query43	541	521	493	493
query44	1325	862	856	856
query45	189	176	168	168
query46	880	1068	662	662
query47	1998	1958	1943	1943
query48	382	421	324	324
query49	696	495	400	400
query50	644	670	396	396
query51	7578	7255	7276	7255
query52	97	97	90	90
query53	220	250	181	181
query54	484	493	412	412
query55	82	79	76	76
query56	259	259	287	259
query57	1247	1223	1182	1182
query58	235	222	227	222
query59	3058	3143	3099	3099
query60	274	265	260	260
query61	112	115	108	108
query62	862	832	757	757
query63	236	187	204	187
query64	2827	1005	672	672
query65	3348	3280	3282	3280
query66	737	417	330	330
query67	16750	15820	15456	15456
query68	7894	684	510	510
query69	473	300	252	252
query70	1200	1166	1141	1141
query71	430	283	247	247
query72	6159	3897	3927	3897
query73	638	746	355	355
query74	9730	9358	9091	9091
query75	3915	3156	2681	2681
query76	3566	1191	753	753
query77	749	377	264	264
query78	10066	10600	9653	9653
query79	3350	811	590	590
query80	721	497	439	439
query81	479	266	226	226
query82	467	151	121	121
query83	193	164	152	152
query84	289	92	74	74
query85	743	357	296	296
query86	359	320	316	316
query87	4543	4645	4515	4515
query88	4501	2260	2175	2175
query89	417	330	297	297
query90	1833	190	195	190
query91	132	135	102	102
query92	67	57	52	52
query93	1114	831	528	528
query94	642	385	285	285
query95	337	267	250	250
query96	495	607	278	278
query97	2915	3030	2861	2861
query98	226	215	215	215
query99	1627	1562	1468	1468
Total cold run time: 293916 ms
Total hot run time: 197715 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.6 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0a3cd2502d25b0508440502d19c82e3489892601, data reload: false

query1	0.03	0.06	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.60	0.11	0.11
query5	0.40	0.43	0.41
query6	1.16	0.67	0.66
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.59	0.50	0.50
query10	0.55	0.56	0.56
query11	0.14	0.10	0.11
query12	0.15	0.12	0.12
query13	0.60	0.60	0.59
query14	2.85	2.74	2.88
query15	0.91	0.82	0.83
query16	0.39	0.40	0.39
query17	1.02	1.05	1.03
query18	0.23	0.21	0.20
query19	1.84	1.71	1.99
query20	0.01	0.01	0.02
query21	15.36	0.91	0.55
query22	0.75	0.78	0.66
query23	15.36	1.44	0.51
query24	3.07	0.40	1.64
query25	0.22	0.16	0.05
query26	0.29	0.15	0.12
query27	0.06	0.05	0.06
query28	13.74	1.53	1.04
query29	12.57	3.90	3.32
query30	0.25	0.09	0.06
query31	2.85	0.58	0.37
query32	3.23	0.55	0.47
query33	3.09	3.23	3.08
query34	16.69	5.19	4.52
query35	4.51	4.50	4.50
query36	0.82	0.48	0.51
query37	0.10	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.03
query40	0.17	0.12	0.12
query41	0.09	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.21 s
Total hot run time: 30.6 s

@BePPPower
Copy link
Contributor Author

run buildall

morningman
morningman reviewed Jan 7, 2025
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSDlaTable.java
import java.util.Optional;
import java.util.Set;

public abstract class HMSDlaTable implements MTMVBaseTableIf {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add some comment for this class.
eg, why need this class, what's is purpose?

fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalTable.java Outdated Show resolved Hide resolved
@BePPPower BePPPower force-pushed the implementHiveExternalTable2 branch from 931bba6 to 95df08a Compare January 7, 2025 07:03
@BePPPower
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32791 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 56ad2bbffb39612807f46478460dda31cc2c5cd9, data reload: false

------ Round 1 ----------------------------------
q1	17605	6129	6019	6019
q2	2049	306	164	164
q3	10425	1235	758	758
q4	10215	885	434	434
q5	7589	2175	1995	1995
q6	205	181	148	148
q7	897	762	603	603
q8	9241	1366	1179	1179
q9	5295	5023	5011	5011
q10	6810	2324	1894	1894
q11	495	290	262	262
q12	348	370	224	224
q13	18159	3646	3068	3068
q14	227	233	208	208
q15	564	508	511	508
q16	618	636	576	576
q17	555	837	322	322
q18	6804	6412	6466	6412
q19	1698	957	540	540
q20	304	316	186	186
q21	2821	2238	1975	1975
q22	363	333	305	305
Total cold run time: 103287 ms
Total hot run time: 32791 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6214	6196	6198	6196
q2	232	332	239	239
q3	2224	2621	2303	2303
q4	1470	1877	1403	1403
q5	4347	4736	4761	4736
q6	183	176	141	141
q7	2059	1955	1782	1782
q8	2618	2807	2685	2685
q9	7348	7278	7413	7278
q10	3067	3359	2858	2858
q11	568	499	501	499
q12	634	739	611	611
q13	3557	3919	3175	3175
q14	284	317	295	295
q15	561	519	521	519
q16	667	672	654	654
q17	1201	1732	1231	1231
q18	7637	7572	7395	7395
q19	820	926	1124	926
q20	2016	2035	1916	1916
q21	5586	5303	4806	4806
q22	610	625	586	586
Total cold run time: 53903 ms
Total hot run time: 52234 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198978 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 56ad2bbffb39612807f46478460dda31cc2c5cd9, data reload: false

query1	1320	957	911	911
query2	6488	2331	2339	2331
query3	11102	5007	4871	4871
query4	32974	23752	24061	23752
query5	3716	618	461	461
query6	277	215	186	186
query7	3982	492	307	307
query8	315	250	242	242
query9	9361	2774	2762	2762
query10	461	328	248	248
query11	17838	15821	15606	15606
query12	159	110	105	105
query13	1738	544	406	406
query14	11228	8038	7810	7810
query15	246	208	190	190
query16	8138	609	506	506
query17	1589	774	632	632
query18	2138	409	320	320
query19	193	182	160	160
query20	159	130	115	115
query21	204	121	101	101
query22	4704	4769	4642	4642
query23	34904	35003	33748	33748
query24	6028	2333	2276	2276
query25	449	459	382	382
query26	800	273	154	154
query27	2044	467	337	337
query28	5306	2556	2488	2488
query29	534	552	431	431
query30	211	190	168	168
query31	983	921	822	822
query32	90	58	59	58
query33	513	338	292	292
query34	761	864	528	528
query35	804	831	785	785
query36	1011	1033	967	967
query37	124	104	81	81
query38	4234	4599	4460	4460
query39	1541	1457	1474	1457
query40	196	122	101	101
query41	46	43	45	43
query42	118	104	142	104
query43	521	515	498	498
query44	1351	823	819	819
query45	181	175	169	169
query46	889	1063	666	666
query47	2025	2035	1961	1961
query48	409	412	349	349
query49	719	495	410	410
query50	644	655	380	380
query51	7176	7144	6957	6957
query52	103	98	89	89
query53	229	252	192	192
query54	465	507	414	414
query55	76	79	77	77
query56	271	269	245	245
query57	1263	1266	1176	1176
query58	240	235	237	235
query59	3117	3330	3267	3267
query60	280	268	247	247
query61	107	106	131	106
query62	825	860	759	759
query63	244	188	189	188
query64	3634	1038	675	675
query65	3327	3259	3276	3259
query66	769	410	306	306
query67	16363	15938	15619	15619
query68	9540	699	506	506
query69	452	294	250	250
query70	1220	1150	1111	1111
query71	448	283	253	253
query72	6142	3929	3820	3820
query73	653	763	375	375
query74	10112	9121	8889	8889
query75	4438	3150	2645	2645
query76	4054	1204	770	770
query77	786	372	281	281
query78	10222	9968	9450	9450
query79	4066	783	573	573
query80	712	595	430	430
query81	492	276	231	231
query82	614	154	128	128
query83	196	162	142	142
query84	277	87	74	74
query85	766	380	293	293
query86	397	312	306	306
query87	4455	4359	4405	4359
query88	4648	2214	2262	2214
query89	419	328	306	306
query90	1803	185	184	184
query91	128	132	104	104
query92	63	54	52	52
query93	2458	860	521	521
query94	629	390	261	261
query95	322	260	268	260
query96	492	611	276	276
query97	2943	2994	2875	2875
query98	224	196	197	196
query99	1702	1542	1434	1434
Total cold run time: 300128 ms
Total hot run time: 198978 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.83 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 56ad2bbffb39612807f46478460dda31cc2c5cd9, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.06	0.07
query4	1.61	0.11	0.10
query5	0.40	0.43	0.42
query6	1.15	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.52	0.53
query10	0.56	0.57	0.55
query11	0.14	0.10	0.10
query12	0.14	0.10	0.10
query13	0.60	0.61	0.60
query14	2.72	2.72	2.80
query15	0.89	0.84	0.84
query16	0.38	0.38	0.38
query17	1.06	1.01	1.05
query18	0.23	0.21	0.20
query19	1.97	1.86	1.90
query20	0.02	0.01	0.02
query21	15.36	0.91	0.55
query22	0.75	0.85	0.71
query23	15.17	1.43	0.53
query24	2.70	1.24	1.53
query25	0.20	0.15	0.10
query26	0.33	0.14	0.13
query27	0.06	0.05	0.05
query28	13.77	1.57	1.04
query29	12.57	4.02	3.38
query30	0.25	0.09	0.07
query31	2.83	0.58	0.38
query32	3.22	0.53	0.47
query33	3.12	3.06	3.12
query34	16.87	5.15	4.54
query35	4.52	4.61	4.58
query36	0.64	0.50	0.47
query37	0.10	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.14
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.03	0.02	0.03
Total cold run time: 105.69 s
Total hot run time: 31.83 s

@BePPPower
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33015 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e21a13b53abd9711db7b95336c4c9674c75b4e17, data reload: false

------ Round 1 ----------------------------------
q1	17573	6202	6128	6128
q2	2049	312	178	178
q3	10509	1242	767	767
q4	10227	865	437	437
q5	8001	2211	2002	2002
q6	213	186	153	153
q7	905	761	607	607
q8	9229	1372	1184	1184
q9	5215	4919	4914	4914
q10	6757	2292	1833	1833
q11	505	288	268	268
q12	350	352	220	220
q13	17777	3748	3093	3093
q14	235	243	210	210
q15	567	511	490	490
q16	635	605	580	580
q17	588	849	338	338
q18	6985	6601	6573	6573
q19	2274	965	560	560
q20	296	310	185	185
q21	2945	2206	1990	1990
q22	356	333	305	305
Total cold run time: 104191 ms
Total hot run time: 33015 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6289	6219	6213	6213
q2	228	327	230	230
q3	2238	2624	2282	2282
q4	1421	1814	1410	1410
q5	4358	4741	4829	4741
q6	184	174	141	141
q7	2071	1970	1811	1811
q8	2682	2783	2702	2702
q9	7304	7296	7354	7296
q10	3098	3334	2816	2816
q11	570	502	492	492
q12	645	742	593	593
q13	3505	3896	3207	3207
q14	286	312	280	280
q15	548	522	524	522
q16	646	681	654	654
q17	1224	1716	1247	1247
q18	7719	7522	7310	7310
q19	822	1222	1056	1056
q20	2031	2069	1919	1919
q21	5797	5282	4808	4808
q22	615	635	595	595
Total cold run time: 54281 ms
Total hot run time: 52325 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198253 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e21a13b53abd9711db7b95336c4c9674c75b4e17, data reload: false

query1	1306	921	919	919
query2	6209	2252	2324	2252
query3	11006	4645	4771	4645
query4	33047	23941	23443	23443
query5	4517	629	470	470
query6	306	199	195	195
query7	4001	488	312	312
query8	287	230	227	227
query9	9379	2766	2756	2756
query10	459	309	267	267
query11	18190	15396	15147	15147
query12	156	108	103	103
query13	1588	545	426	426
query14	10903	6899	7667	6899
query15	267	225	188	188
query16	7846	600	426	426
query17	1534	769	586	586
query18	1948	401	329	329
query19	216	191	161	161
query20	127	114	109	109
query21	205	125	109	109
query22	4684	4685	4521	4521
query23	34338	34291	33818	33818
query24	6357	2353	2317	2317
query25	467	471	394	394
query26	774	271	156	156
query27	2066	467	344	344
query28	5200	2546	2546	2546
query29	536	548	432	432
query30	214	180	148	148
query31	1002	913	880	880
query32	85	62	57	57
query33	497	352	339	339
query34	761	866	518	518
query35	833	845	777	777
query36	1044	1064	997	997
query37	116	99	72	72
query38	4318	4300	4145	4145
query39	1567	1479	1444	1444
query40	208	115	99	99
query41	43	43	43	43
query42	127	107	105	105
query43	531	534	486	486
query44	1360	839	845	839
query45	188	172	165	165
query46	908	1067	683	683
query47	2038	2055	2010	2010
query48	372	445	322	322
query49	725	467	399	399
query50	633	694	400	400
query51	6989	7122	7111	7111
query52	110	103	89	89
query53	225	258	186	186
query54	478	531	410	410
query55	84	79	86	79
query56	269	275	244	244
query57	1243	1268	1181	1181
query58	247	227	230	227
query59	3106	3369	3264	3264
query60	288	265	259	259
query61	123	106	117	106
query62	859	826	740	740
query63	235	197	190	190
query64	3747	1059	661	661
query65	3345	3272	3227	3227
query66	923	420	322	322
query67	16684	15869	15695	15695
query68	8609	700	523	523
query69	465	297	263	263
query70	1206	1149	1117	1117
query71	450	295	257	257
query72	6247	3911	3873	3873
query73	644	759	378	378
query74	10131	9245	9096	9096
query75	4068	3161	2666	2666
query76	3630	1181	773	773
query77	763	359	305	305
query78	10208	10238	10256	10238
query79	3652	812	586	586
query80	635	517	422	422
query81	472	287	232	232
query82	600	149	124	124
query83	196	159	142	142
query84	273	84	79	79
query85	758	366	336	336
query86	355	323	286	286
query87	4429	4515	4497	4497
query88	4617	2358	2226	2226
query89	423	340	297	297
query90	1896	187	185	185
query91	134	135	109	109
query92	62	56	57	56
query93	1796	852	540	540
query94	642	405	294	294
query95	349	272	263	263
query96	484	615	285	285
query97	2879	2995	2780	2780
query98	216	199	194	194
query99	1645	1503	1415	1415
Total cold run time: 297378 ms
Total hot run time: 198253 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.03 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e21a13b53abd9711db7b95336c4c9674c75b4e17, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.11
query5	0.41	0.42	0.41
query6	1.17	0.65	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.52	0.50
query10	0.55	0.57	0.55
query11	0.15	0.11	0.10
query12	0.13	0.11	0.10
query13	0.60	0.59	0.60
query14	2.80	2.81	2.77
query15	0.89	0.82	0.82
query16	0.39	0.39	0.38
query17	1.07	1.02	1.07
query18	0.23	0.21	0.20
query19	1.88	1.89	2.00
query20	0.02	0.01	0.01
query21	15.39	0.93	0.57
query22	0.75	0.90	0.86
query23	14.94	1.46	0.51
query24	3.27	1.39	1.65
query25	0.13	0.22	0.14
query26	0.25	0.15	0.14
query27	0.06	0.04	0.04
query28	14.39	1.47	1.05
query29	12.54	4.02	3.31
query30	0.25	0.09	0.07
query31	2.80	0.59	0.39
query32	3.23	0.54	0.44
query33	3.10	3.08	3.06
query34	16.83	5.09	4.52
query35	4.46	4.47	4.50
query36	0.83	0.47	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.63 s
Total hot run time: 32.03 s

@BePPPower
Copy link
Contributor Author

run p0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@BePPPower @doris-robot @morningman