[Enhancement] Optimize performance of arrays_overlaping (backport #49668) #51561

mergify · 2024-09-30T09:24:04Z

Why I'm doing:

Datum is not an efficient data structure, and arrrasy_overlapping heavily relies on Datum for processing, resulting in poor performance.

The second parameter of arrrasy_overlapping is usually const, but there is currently no special handling.

What I'm doing:

PerformaceTest: 27.76s -> 2.74s

mysql> select count(*) from lineorder;
+-----------+
| count(*)  |
+-----------+
| 143999468 |
+-----------+
1 row in set (5.17 sec)

CREATE TABLE `t2` (
  `c1` bigint(20) NOT NULL COMMENT "",
  `c2` array<varchar(65533)> NOT NULL COMMENT "",
  `c3` array<varchar(65533)> NOT NULL COMMENT ""
) ENGINE=OLAP 
DUPLICATE KEY(`c1`)
DISTRIBUTED BY HASH(`c1`) BUCKETS 1 
PROPERTIES (
"compression" = "LZ4",
"fast_schema_evolution" = "true",
"replicated_storage" = "true",
"replication_num" = "1"
);

insert into t2 select lo_orderkey, array_agg(lo_partkey), array_agg(lo_suppkey) from lineorder group by lo_orderkey;

mysql> select count(*) from t2;
+----------+
| count(*) |
+----------+
| 30000000 |
+----------+
1 row in set (0.36 sec)

Before optimization: 27.76s

mysql> select count(*) from t2 where arrays_overlap(c2, ["1", "11", "111", "1111", "11111", "111111", "1111111"]);                                                                                                                                                                
+----------+                                                                                                                                                                                                                                                                      
| count(*) |                                                                                                                                                                                                                                                                      
+----------+                                                                                                                                                                                                                                                                      
|     1204 |                                                                                                                                                                                                                                                                      
+----------+                                                                                                                                                                                                                                                                      
1 row in set (27.76 sec)

After optimization: 2.74s

mysql> select count(*) from t2 where arrays_overlap(c2, ["1", "11", "111", "1111", "11111", "111111", "1111111"]);                                                                                                                                                                
+----------+                                                                                                                                                                                                                                                                      
| count(*) |                                                                                                                                                                                                                                                                      
+----------+                                                                                                                                                                                                                                                                      
|     1204 |                                                                                                                                                                                                                                                                      
+----------+                                                                                                                                                                                                                                                                      
1 row in set (2.74 sec)

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

Documentation PRs only:

If you are submitting a PR that adds or changes English documentation and have not
included Chinese documentation, then you can check the box to request GPT to translate the
English doc to Chinese. Please ensure to uncheck the Do not translate box if translation is needed.
The workflow will generate a new PR with the Chinese translation after this PR is merged.

Yes, translate English markdown files with GPT
Do not translate

This is an automatic backport of pull request #49668 done by [Mergify](https://mergify.com). ## Why I'm doing:

Datum is not an efficient data structure, and arrrasy_overlapping heavily relies on Datum for processing, resulting in poor performance.

The second parameter of arrrasy_overlapping is usually const, but there is currently no special handling.

What I'm doing:

PerformaceTest: 27.76s -> 2.74s

mysql> select count(*) from lineorder;
+-----------+
| count(*)  |
+-----------+
| 143999468 |
+-----------+
1 row in set (5.17 sec)

CREATE TABLE `t2` (
  `c1` bigint(20) NOT NULL COMMENT "",
  `c2` array<varchar(65533)> NOT NULL COMMENT "",
  `c3` array<varchar(65533)> NOT NULL COMMENT ""
) ENGINE=OLAP 
DUPLICATE KEY(`c1`)
DISTRIBUTED BY HASH(`c1`) BUCKETS 1 
PROPERTIES (
"compression" = "LZ4",
"fast_schema_evolution" = "true",
"replicated_storage" = "true",
"replication_num" = "1"
);

insert into t2 select lo_orderkey, array_agg(lo_partkey), array_agg(lo_suppkey) from lineorder group by lo_orderkey;

mysql> select count(*) from t2;
+----------+
| count(*) |
+----------+
| 30000000 |
+----------+
1 row in set (0.36 sec)

Before optimization: 27.76s

mysql> select count(*) from t2 where arrays_overlap(c2, ["1", "11", "111", "1111", "11111", "111111", "1111111"]);                                                                                                                                                                
+----------+                                                                                                                                                                                                                                                                      
| count(*) |                                                                                                                                                                                                                                                                      
+----------+                                                                                                                                                                                                                                                                      
|     1204 |                                                                                                                                                                                                                                                                      
+----------+                                                                                                                                                                                                                                                                      
1 row in set (27.76 sec)

After optimization: 2.74s

mysql> select count(*) from t2 where arrays_overlap(c2, ["1", "11", "111", "1111", "11111", "111111", "1111111"]);                                                                                                                                                                
+----------+                                                                                                                                                                                                                                                                      
| count(*) |                                                                                                                                                                                                                                                                      
+----------+                                                                                                                                                                                                                                                                      
|     1204 |                                                                                                                                                                                                                                                                      
+----------+                                                                                                                                                                                                                                                                      
1 row in set (2.74 sec)

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Signed-off-by: trueeyu <[email protected]> (cherry picked from commit 20d8075) # Conflicts: # be/src/exprs/array_functions.tpp

mergify · 2024-09-30T09:24:05Z

Cherry-pick of 20d8075 has failed:

On branch mergify/bp/branch-3.1/pr-49668
Your branch is up to date with 'origin/branch-3.1'.

You are currently cherry-picking commit 20d8075551.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   be/src/column/column_helper.h
	modified:   be/src/exprs/array_functions.h
	modified:   be/src/exprs/function_context.cpp
	modified:   be/test/column/column_helper_test.cpp
	modified:   be/test/exprs/array_functions_test.cpp
	modified:   gensrc/script/functions.py
	modified:   test/sql/test_array_fn/R/test_array_fn
	new file:   test/sql/test_array_fn/R/test_arrays_overlap
	modified:   test/sql/test_array_fn/T/test_array_fn
	new file:   test/sql/test_array_fn/T/test_arrays_overlap

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   be/src/exprs/array_functions.tpp

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

mergify · 2024-09-30T09:24:42Z

@mergify[bot]: Backport conflict, please reslove the conflict and resubmit the pr

Signed-off-by: trueeyu <[email protected]>

sonarcloud · 2024-09-30T09:56:18Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[Enhancement] Optimize performance of arrays_overlaping (#49668)

8a2a62d

Signed-off-by: trueeyu <[email protected]> (cherry picked from commit 20d8075) # Conflicts: # be/src/exprs/array_functions.tpp

mergify bot requested a review from a team as a code owner September 30, 2024 09:24

mergify bot added the conflicts label Sep 30, 2024

mergify bot mentioned this pull request Sep 30, 2024

[Enhancement] Optimize performance of arrays_overlaping #49668

Merged

26 tasks

github-actions bot assigned trueeyu Sep 30, 2024

mergify bot closed this Sep 30, 2024

github-actions bot added the automerge label Sep 30, 2024

trueeyu reopened this Sep 30, 2024

wanpengfei-git enabled auto-merge (squash) September 30, 2024 09:29

trueeyu added 2 commits September 30, 2024 17:41

fix

45b4349

Signed-off-by: trueeyu <[email protected]>

fix

2749f5c

Signed-off-by: trueeyu <[email protected]>

trueeyu approved these changes Sep 30, 2024

View reviewed changes

wanpengfei-git merged commit 146a897 into branch-3.1 Sep 30, 2024
29 checks passed

wanpengfei-git deleted the mergify/bp/branch-3.1/pr-49668 branch September 30, 2024 10:13

github-actions bot added the version:3.1.16 label Sep 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Optimize performance of arrays_overlaping (backport #49668) #51561

[Enhancement] Optimize performance of arrays_overlaping (backport #49668) #51561

mergify bot commented Sep 30, 2024 •

edited by wanpengfei-git

Loading

mergify bot commented Sep 30, 2024

mergify bot commented Sep 30, 2024

sonarcloud bot commented Sep 30, 2024

[Enhancement] Optimize performance of arrays_overlaping (backport #49668) #51561

[Enhancement] Optimize performance of arrays_overlaping (backport #49668) #51561

Conversation

mergify bot commented Sep 30, 2024 • edited by wanpengfei-git Loading

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

Documentation PRs only:

What I'm doing:

What type of PR is this:

Checklist:

mergify bot commented Sep 30, 2024

mergify bot commented Sep 30, 2024

sonarcloud bot commented Sep 30, 2024

Quality Gate passed

mergify bot commented Sep 30, 2024 •

edited by wanpengfei-git

Loading