Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](nereids) fix ptopN push down under multi winexprs with partial forbidden type #44617

Merged
merged 2 commits into from
Nov 27, 2024

Conversation

xzj7019
Copy link
Contributor

@xzj7019 xzj7019 commented Nov 26, 2024

What problem does this PR solve?

Related PR: #38393

Problem Summary: In the previous pr which supporting multi win expr ptopN pushdown, it handled partial forbidden type unexpectly and will lead some case to push down the pTopN wrongly.
plan before fixing:

explain shape plan select * from (select row_number() over(partition by c1, c2 order by c3) as rn, sum(c2) over(order by c2 range between unbounded preceding and unbounded following) as sw from push_down_multi_predicate_through_window_t) t where rn <= 1 and sw <= 1;
+------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                    |
+------------------------------------------------------------------------------------+
| PhysicalResultSink                                                                 |
| --PhysicalProject                                                                  |
| ----filter((rn <= 1) and (sw <= 1))                                                |
| ------PhysicalWindow                                                               |
| --------PhysicalQuickSort[MERGE_SORT]                                              |
| ----------PhysicalDistribute[DistributionSpecGather]                               |
| ------------PhysicalQuickSort[LOCAL_SORT]                                          |
| --------------PhysicalWindow                                                       |
| ----------------PhysicalQuickSort[LOCAL_SORT]                                      |
| ------------------PhysicalDistribute[DistributionSpecHash]                         |
| --------------------PhysicalPartitionTopN                                          |
| ----------------------PhysicalOlapScan[push_down_multi_predicate_through_window_t] |
+------------------------------------------------------------------------------------+

plan after fixing:

explain shape plan select * from (select row_number() over(partition by c1, c2 order by c3) as rn, sum(c2) over(order by c2 range between unbounded preceding and unbounded following) as sw from push_down_multi_predicate_through_window_t) t where rn <= 1 and sw <= 1;
+----------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                  |
+----------------------------------------------------------------------------------+
| PhysicalResultSink                                                               |
| --PhysicalProject                                                                |
| ----filter((rn <= 1) and (sw <= 1))                                              |
| ------PhysicalWindow                                                             |
| --------PhysicalQuickSort[MERGE_SORT]                                            |
| ----------PhysicalDistribute[DistributionSpecGather]                             |
| ------------PhysicalQuickSort[LOCAL_SORT]                                        |
| --------------PhysicalWindow                                                     |
| ----------------PhysicalQuickSort[LOCAL_SORT]                                    |
| ------------------PhysicalDistribute[DistributionSpecHash]                       |
| --------------------PhysicalOlapScan[push_down_multi_predicate_through_window_t] |
+----------------------------------------------------------------------------------+

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xzj7019
Copy link
Contributor Author

xzj7019 commented Nov 26, 2024

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 26, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 40263 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3014a68077cda0ab3b48af810f3c69ffe67e6b33, data reload: false

------ Round 1 ----------------------------------
q1	17591	7824	7410	7410
q2	2047	179	170	170
q3	10649	1135	1198	1135
q4	10460	737	720	720
q5	7599	2742	2701	2701
q6	245	154	145	145
q7	995	626	612	612
q8	9245	1860	1900	1860
q9	6543	6420	6483	6420
q10	7003	2277	2312	2277
q11	457	259	256	256
q12	423	218	214	214
q13	17779	3045	3020	3020
q14	246	214	222	214
q15	566	526	514	514
q16	659	597	598	597
q17	980	619	546	546
q18	7387	6747	6801	6747
q19	1338	1006	937	937
q20	483	179	181	179
q21	4024	3329	3273	3273
q22	382	319	316	316
Total cold run time: 107101 ms
Total hot run time: 40263 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7432	7337	7286	7286
q2	323	233	238	233
q3	2952	2885	2992	2885
q4	2075	1851	1837	1837
q5	5653	5650	5664	5650
q6	224	141	140	140
q7	2277	1819	1825	1819
q8	3459	3551	3524	3524
q9	8777	8948	8953	8948
q10	3650	3587	3624	3587
q11	606	507	510	507
q12	807	612	644	612
q13	12431	3244	3218	3218
q14	326	272	291	272
q15	571	527	527	527
q16	699	667	652	652
q17	1871	1632	1578	1578
q18	8403	7983	7717	7717
q19	1739	1561	1484	1484
q20	2095	1838	1941	1838
q21	5720	5514	5534	5514
q22	674	565	551	551
Total cold run time: 72764 ms
Total hot run time: 60379 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196991 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3014a68077cda0ab3b48af810f3c69ffe67e6b33, data reload: false

query1	1235	951	930	930
query2	6400	2021	1970	1970
query3	10897	4105	4048	4048
query4	67749	28324	23426	23426
query5	5145	483	472	472
query6	436	203	176	176
query7	5737	303	292	292
query8	308	223	227	223
query9	9079	2638	2637	2637
query10	482	260	253	253
query11	17320	15362	15996	15362
query12	154	106	114	106
query13	1587	432	427	427
query14	11299	7547	7628	7547
query15	207	194	187	187
query16	7201	501	498	498
query17	1103	588	594	588
query18	1909	301	296	296
query19	211	154	158	154
query20	119	113	113	113
query21	215	99	101	99
query22	4774	4591	4467	4467
query23	34655	34271	34379	34271
query24	5456	2540	2491	2491
query25	486	394	391	391
query26	650	152	150	150
query27	1833	286	287	286
query28	4296	2462	2489	2462
query29	662	412	432	412
query30	205	154	155	154
query31	1009	823	856	823
query32	64	58	53	53
query33	421	280	283	280
query34	931	521	537	521
query35	826	775	753	753
query36	1102	952	933	933
query37	116	69	81	69
query38	4522	4452	4424	4424
query39	1504	1452	1468	1452
query40	203	103	100	100
query41	45	43	44	43
query42	112	102	99	99
query43	528	498	500	498
query44	1181	855	838	838
query45	187	170	170	170
query46	1157	732	716	716
query47	2018	1943	1915	1915
query48	415	319	310	310
query49	757	390	399	390
query50	830	400	393	393
query51	7284	7195	7100	7100
query52	97	92	86	86
query53	258	183	184	183
query54	502	395	397	395
query55	81	76	78	76
query56	258	252	224	224
query57	1269	1210	1162	1162
query58	228	241	224	224
query59	3324	2891	2852	2852
query60	263	246	238	238
query61	111	111	110	110
query62	768	673	665	665
query63	216	188	189	188
query64	1370	665	650	650
query65	3259	3212	3231	3212
query66	704	294	295	294
query67	15958	15900	15698	15698
query68	4006	572	571	571
query69	421	251	255	251
query70	1203	1114	1103	1103
query71	346	249	245	245
query72	6452	4146	4070	4070
query73	757	358	360	358
query74	10128	9046	9022	9022
query75	3366	2680	2700	2680
query76	1979	1006	1085	1006
query77	518	284	277	277
query78	10452	9661	9378	9378
query79	2127	599	594	594
query80	1313	423	525	423
query81	520	243	232	232
query82	847	119	120	119
query83	232	149	150	149
query84	272	78	71	71
query85	1012	307	294	294
query86	399	309	313	309
query87	4723	4551	4786	4551
query88	3366	2178	2159	2159
query89	411	290	339	290
query90	2058	189	188	188
query91	142	103	105	103
query92	71	51	50	50
query93	2400	544	544	544
query94	915	295	291	291
query95	352	254	252	252
query96	618	279	275	275
query97	2867	2639	2667	2639
query98	220	192	195	192
query99	1808	1280	1299	1280
Total cold run time: 321989 ms
Total hot run time: 196991 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3014a68077cda0ab3b48af810f3c69ffe67e6b33, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.09	0.07
query4	1.61	0.11	0.10
query5	0.44	0.42	0.41
query6	1.14	0.66	0.64
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.56	0.54	0.51
query10	0.56	0.55	0.55
query11	0.15	0.11	0.10
query12	0.14	0.12	0.11
query13	0.63	0.61	0.60
query14	2.83	2.77	2.85
query15	0.93	0.85	0.82
query16	0.40	0.39	0.37
query17	1.09	1.02	1.07
query18	0.22	0.20	0.20
query19	1.83	1.79	1.97
query20	0.02	0.00	0.02
query21	15.38	0.63	0.59
query22	2.36	1.73	2.02
query23	17.00	0.88	0.89
query24	3.15	1.79	1.47
query25	0.37	0.04	0.05
query26	0.65	0.13	0.13
query27	0.04	0.06	0.03
query28	9.49	1.11	1.07
query29	12.60	3.24	3.19
query30	0.24	0.06	0.06
query31	2.87	0.38	0.38
query32	3.27	0.46	0.47
query33	3.04	3.02	3.13
query34	17.01	4.51	4.48
query35	4.61	4.52	4.49
query36	0.67	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.17 s
Total hot run time: 33 s

@morrySnow morrySnow added usercase Important user case type label p0_w labels Nov 26, 2024
@morrySnow morrySnow merged commit aca0eaa into apache:master Nov 27, 2024
36 of 38 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 27, 2024
…forbidden type (#44617)

Related PR: #38393

Problem Summary: In the previous pr which supporting multi win expr
ptopN pushdown, it handled partial forbidden type unexpectly and will
lead some case to push down the pTopN wrongly.
plan before fixing:

explain shape plan select * from (select row_number() over(partition by c1, c2 order by c3) as rn, sum(c2) over(order by c2 range between unbounded preceding and unbounded following) as sw from push_down_multi_predicate_through_window_t) t where rn <= 1 and sw <= 1;

+------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                    |
+------------------------------------------------------------------------------------+
| PhysicalResultSink                                                                 |
| --PhysicalProject                                                                  |
| ----filter((rn <= 1) and (sw <= 1))                                                |
| ------PhysicalWindow                                                               |
| --------PhysicalQuickSort[MERGE_SORT]                                              |
| ----------PhysicalDistribute[DistributionSpecGather]                               |
| ------------PhysicalQuickSort[LOCAL_SORT]                                          |
| --------------PhysicalWindow                                                       |
| ----------------PhysicalQuickSort[LOCAL_SORT]                                      |
| ------------------PhysicalDistribute[DistributionSpecHash]                         |
| --------------------PhysicalPartitionTopN                                          |
| ----------------------PhysicalOlapScan[push_down_multi_predicate_through_window_t] |
+------------------------------------------------------------------------------------+

plan after fixing: 

explain shape plan select * from (select row_number() over(partition by c1, c2 order by c3) as rn, sum(c2) over(order by c2 range between unbounded preceding and unbounded following) as sw from push_down_multi_predicate_through_window_t) t where rn <= 1 and sw <= 1;

+----------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                  |
+----------------------------------------------------------------------------------+
| PhysicalResultSink                                                               |
| --PhysicalProject                                                                |
| ----filter((rn <= 1) and (sw <= 1))                                              |
| ------PhysicalWindow                                                             |
| --------PhysicalQuickSort[MERGE_SORT]                                            |
| ----------PhysicalDistribute[DistributionSpecGather]                             |
| ------------PhysicalQuickSort[LOCAL_SORT]                                        |
| --------------PhysicalWindow                                                     |
| ----------------PhysicalQuickSort[LOCAL_SORT]                                    |
| ------------------PhysicalDistribute[DistributionSpecHash]                       |
| --------------------PhysicalOlapScan[push_down_multi_predicate_through_window_t] |
+----------------------------------------------------------------------------------+
github-actions bot pushed a commit that referenced this pull request Nov 27, 2024
…forbidden type (#44617)

Related PR: #38393

Problem Summary: In the previous pr which supporting multi win expr
ptopN pushdown, it handled partial forbidden type unexpectly and will
lead some case to push down the pTopN wrongly.
plan before fixing:

explain shape plan select * from (select row_number() over(partition by c1, c2 order by c3) as rn, sum(c2) over(order by c2 range between unbounded preceding and unbounded following) as sw from push_down_multi_predicate_through_window_t) t where rn <= 1 and sw <= 1;

+------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                    |
+------------------------------------------------------------------------------------+
| PhysicalResultSink                                                                 |
| --PhysicalProject                                                                  |
| ----filter((rn <= 1) and (sw <= 1))                                                |
| ------PhysicalWindow                                                               |
| --------PhysicalQuickSort[MERGE_SORT]                                              |
| ----------PhysicalDistribute[DistributionSpecGather]                               |
| ------------PhysicalQuickSort[LOCAL_SORT]                                          |
| --------------PhysicalWindow                                                       |
| ----------------PhysicalQuickSort[LOCAL_SORT]                                      |
| ------------------PhysicalDistribute[DistributionSpecHash]                         |
| --------------------PhysicalPartitionTopN                                          |
| ----------------------PhysicalOlapScan[push_down_multi_predicate_through_window_t] |
+------------------------------------------------------------------------------------+

plan after fixing: 

explain shape plan select * from (select row_number() over(partition by c1, c2 order by c3) as rn, sum(c2) over(order by c2 range between unbounded preceding and unbounded following) as sw from push_down_multi_predicate_through_window_t) t where rn <= 1 and sw <= 1;

+----------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                  |
+----------------------------------------------------------------------------------+
| PhysicalResultSink                                                               |
| --PhysicalProject                                                                |
| ----filter((rn <= 1) and (sw <= 1))                                              |
| ------PhysicalWindow                                                             |
| --------PhysicalQuickSort[MERGE_SORT]                                            |
| ----------PhysicalDistribute[DistributionSpecGather]                             |
| ------------PhysicalQuickSort[LOCAL_SORT]                                        |
| --------------PhysicalWindow                                                     |
| ----------------PhysicalQuickSort[LOCAL_SORT]                                    |
| ------------------PhysicalDistribute[DistributionSpecHash]                       |
| --------------------PhysicalOlapScan[push_down_multi_predicate_through_window_t] |
+----------------------------------------------------------------------------------+
morrySnow pushed a commit that referenced this pull request Nov 27, 2024
…ith partial forbidden type #44617 (#44649)

Cherry-picked from #44617

Co-authored-by: xzj7019 <[email protected]>
morrySnow pushed a commit that referenced this pull request Nov 27, 2024
…forbidden type (#44617)

Related PR: #38393

Problem Summary: In the previous pr which supporting multi win expr
ptopN pushdown, it handled partial forbidden type unexpectly and will
lead some case to push down the pTopN wrongly.
plan before fixing:

explain shape plan select * from (select row_number() over(partition by c1, c2 order by c3) as rn, sum(c2) over(order by c2 range between unbounded preceding and unbounded following) as sw from push_down_multi_predicate_through_window_t) t where rn <= 1 and sw <= 1;

+------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                    |
+------------------------------------------------------------------------------------+
| PhysicalResultSink                                                                 |
| --PhysicalProject                                                                  |
| ----filter((rn <= 1) and (sw <= 1))                                                |
| ------PhysicalWindow                                                               |
| --------PhysicalQuickSort[MERGE_SORT]                                              |
| ----------PhysicalDistribute[DistributionSpecGather]                               |
| ------------PhysicalQuickSort[LOCAL_SORT]                                          |
| --------------PhysicalWindow                                                       |
| ----------------PhysicalQuickSort[LOCAL_SORT]                                      |
| ------------------PhysicalDistribute[DistributionSpecHash]                         |
| --------------------PhysicalPartitionTopN                                          |
| ----------------------PhysicalOlapScan[push_down_multi_predicate_through_window_t] |
+------------------------------------------------------------------------------------+

plan after fixing: 

explain shape plan select * from (select row_number() over(partition by c1, c2 order by c3) as rn, sum(c2) over(order by c2 range between unbounded preceding and unbounded following) as sw from push_down_multi_predicate_through_window_t) t where rn <= 1 and sw <= 1;

+----------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                  |
+----------------------------------------------------------------------------------+
| PhysicalResultSink                                                               |
| --PhysicalProject                                                                |
| ----filter((rn <= 1) and (sw <= 1))                                              |
| ------PhysicalWindow                                                             |
| --------PhysicalQuickSort[MERGE_SORT]                                            |
| ----------PhysicalDistribute[DistributionSpecGather]                             |
| ------------PhysicalQuickSort[LOCAL_SORT]                                        |
| --------------PhysicalWindow                                                     |
| ----------------PhysicalQuickSort[LOCAL_SORT]                                    |
| ------------------PhysicalDistribute[DistributionSpecHash]                       |
| --------------------PhysicalOlapScan[push_down_multi_predicate_through_window_t] |
+----------------------------------------------------------------------------------+
yiguolei pushed a commit that referenced this pull request Nov 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged p0_w reviewed usercase Important user case type label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants