Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhance](catalog)Unified external partition prune interface #44567

Merged
merged 2 commits into from
Nov 26, 2024

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Nov 25, 2024

What problem does this PR solve?

Previously, external partition cropping only supported Hive. If you want to support other types of tables, you need to understand the internal processing logic of partition pruning.

This PR abstracts the logic of partition pruning, and other tables can be implemented by simply covering a few methods of externalTable

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
Unified external partition prune interface

Release note

Unified external partition prune interface

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zddr
Copy link
Contributor Author

zddr commented Nov 25, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40166 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a9d9d9138b64da073c3eb90d2f3280431fcd0a9d, data reload: false

------ Round 1 ----------------------------------
q1	17885	7586	7264	7264
q2	2048	185	178	178
q3	10622	1147	1171	1147
q4	10260	739	729	729
q5	7615	2759	2664	2664
q6	244	154	150	150
q7	1009	636	625	625
q8	9251	1875	1916	1875
q9	6466	6349	6384	6349
q10	6999	2284	2303	2284
q11	465	265	275	265
q12	436	219	223	219
q13	17772	3071	3023	3023
q14	247	211	211	211
q15	567	520	519	519
q16	635	602	585	585
q17	977	555	587	555
q18	7354	6695	6661	6661
q19	1340	1075	1041	1041
q20	482	181	186	181
q21	4279	3402	3330	3330
q22	393	311	320	311
Total cold run time: 107346 ms
Total hot run time: 40166 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7245	7283	7239	7239
q2	321	219	227	219
q3	2873	2883	3040	2883
q4	2169	1887	1772	1772
q5	5597	5677	5675	5675
q6	224	139	142	139
q7	2205	1798	1890	1798
q8	3421	3522	3486	3486
q9	8890	8901	8796	8796
q10	3612	3558	3540	3540
q11	612	524	513	513
q12	813	639	596	596
q13	12186	3263	3234	3234
q14	310	266	276	266
q15	568	524	526	524
q16	675	642	648	642
q17	1819	1590	1571	1571
q18	7814	7319	7401	7319
q19	1687	1513	1525	1513
q20	2048	1798	1803	1798
q21	5382	5261	5551	5261
q22	637	546	556	546
Total cold run time: 71108 ms
Total hot run time: 59330 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191135 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a9d9d9138b64da073c3eb90d2f3280431fcd0a9d, data reload: false

query1	971	386	382	382
query2	6519	2105	2024	2024
query3	6714	216	211	211
query4	33918	23558	23432	23432
query5	4340	465	443	443
query6	270	182	174	174
query7	4633	300	310	300
query8	284	222	220	220
query9	9370	2730	2737	2730
query10	473	252	256	252
query11	18162	15080	15206	15080
query12	153	104	107	104
query13	1661	438	440	438
query14	10120	7102	7648	7102
query15	306	189	184	184
query16	8212	481	491	481
query17	1833	577	560	560
query18	2143	296	304	296
query19	377	156	153	153
query20	122	112	112	112
query21	211	111	105	105
query22	4573	4205	4441	4205
query23	34823	34209	34142	34142
query24	11520	2427	2512	2427
query25	682	377	384	377
query26	1818	150	151	150
query27	2818	278	279	278
query28	7787	2460	2465	2460
query29	1026	398	410	398
query30	306	150	152	150
query31	1048	811	851	811
query32	98	58	59	58
query33	778	288	280	280
query34	1000	525	523	523
query35	836	720	739	720
query36	1119	938	972	938
query37	282	73	78	73
query38	4426	4271	4174	4174
query39	1481	1449	1474	1449
query40	284	101	96	96
query41	47	44	45	44
query42	107	99	99	99
query43	541	476	492	476
query44	1316	823	815	815
query45	183	162	163	162
query46	1150	712	692	692
query47	1929	1864	1834	1834
query48	417	318	318	318
query49	1298	396	385	385
query50	780	395	388	388
query51	7318	7152	7160	7152
query52	101	95	91	91
query53	264	180	183	180
query54	1297	420	423	420
query55	77	78	80	78
query56	260	266	244	244
query57	1279	1181	1170	1170
query58	232	220	221	220
query59	3123	2908	3044	2908
query60	272	243	253	243
query61	113	129	110	110
query62	890	679	659	659
query63	210	190	186	186
query64	5055	654	635	635
query65	3315	3196	3237	3196
query66	1430	321	309	309
query67	15995	15674	15598	15598
query68	4589	570	587	570
query69	438	253	262	253
query70	1172	1156	1128	1128
query71	350	257	254	254
query72	6402	4040	4004	4004
query73	756	362	360	360
query74	10577	8984	9036	8984
query75	3411	2668	2682	2668
query76	2767	1078	1116	1078
query77	527	358	276	276
query78	10449	9437	9366	9366
query79	1390	594	589	589
query80	1085	447	438	438
query81	547	235	228	228
query82	1009	119	116	116
query83	244	152	151	151
query84	231	75	70	70
query85	1421	294	297	294
query86	394	314	295	295
query87	4730	4644	4493	4493
query88	3383	2278	2211	2211
query89	403	297	300	297
query90	1991	187	184	184
query91	137	101	103	101
query92	61	51	62	51
query93	1078	543	553	543
query94	1114	287	305	287
query95	360	252	252	252
query96	620	288	287	287
query97	2835	2718	2716	2716
query98	218	193	196	193
query99	1596	1340	1294	1294
Total cold run time: 302781 ms
Total hot run time: 191135 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.69 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a9d9d9138b64da073c3eb90d2f3280431fcd0a9d, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.61	0.11	0.10
query5	0.40	0.41	0.41
query6	1.13	0.67	0.66
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.58	0.50	0.52
query10	0.55	0.55	0.56
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.62	0.59
query14	2.70	2.84	2.86
query15	0.91	0.83	0.83
query16	0.38	0.36	0.38
query17	1.08	1.07	1.06
query18	0.23	0.22	0.22
query19	1.84	1.84	2.04
query20	0.02	0.00	0.02
query21	15.36	0.60	0.58
query22	2.67	1.78	1.60
query23	16.93	1.18	0.83
query24	3.31	1.00	1.47
query25	0.11	0.16	0.05
query26	0.66	0.13	0.14
query27	0.05	0.04	0.04
query28	10.43	1.10	1.07
query29	12.55	3.31	3.35
query30	0.26	0.07	0.06
query31	3.34	0.39	0.37
query32	3.35	0.47	0.49
query33	3.03	3.13	3.01
query34	17.43	4.55	4.50
query35	4.57	4.65	4.52
query36	0.69	0.50	0.48
query37	0.09	0.07	0.07
query38	0.04	0.04	0.04
query39	0.04	0.02	0.02
query40	0.15	0.13	0.14
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.94 s
Total hot run time: 32.69 s

@zddr
Copy link
Contributor Author

zddr commented Nov 25, 2024

run cloud_p0

Copy link
Contributor

@hubgeter hubgeter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 26, 2024
@morningman morningman merged commit 25dcd85 into apache:master Nov 26, 2024
34 of 35 checks passed
zddr added a commit to zddr/incubator-doris that referenced this pull request Nov 29, 2024
…44567)

Previously, external partition cropping only supported Hive. If you want
to support other types of tables, you need to understand the internal
processing logic of partition pruning.

This PR abstracts the logic of partition pruning, and other tables can
be implemented by simply covering a few methods of externalTable

[opt](planner) Unified external partition prune interface
hubgeter pushed a commit to hubgeter/doris that referenced this pull request Nov 29, 2024
…44567)

Previously, external partition cropping only supported Hive. If you want
to support other types of tables, you need to understand the internal
processing logic of partition pruning.

This PR abstracts the logic of partition pruning, and other tables can
be implemented by simply covering a few methods of externalTable

[opt](planner) Unified external partition prune interface
zddr added a commit to zddr/incubator-doris that referenced this pull request Dec 2, 2024
…44567)

### What problem does this PR solve?

Previously, external partition cropping only supported Hive. If you want
to support other types of tables, you need to understand the internal
processing logic of partition pruning.

This PR abstracts the logic of partition pruning, and other tables can
be implemented by simply covering a few methods of externalTable

### Release note

[opt](planner) Unified external partition prune interface
zddr added a commit to zddr/incubator-doris that referenced this pull request Dec 6, 2024
…44567)

Previously, external partition cropping only supported Hive. If you want
to support other types of tables, you need to understand the internal
processing logic of partition pruning.

This PR abstracts the logic of partition pruning, and other tables can
be implemented by simply covering a few methods of externalTable

[opt](planner) Unified external partition prune interface
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants