Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feat](nereids) simplify comparison predicate rule add check data type limit #44732

Merged
merged 1 commit into from
Dec 13, 2024
Merged

[feat](nereids) simplify comparison predicate rule add check data type limit #44732

merged 1 commit into from
Dec 13, 2024

Conversation

yujun777
Copy link
Collaborator

@yujun777 yujun777 commented Nov 28, 2024 •
edited
Loading

What problem does this PR solve?

simplify comparison, check data type's limit, like:

suppose a is tinyint, so its range should be [-128, 127], then we can simplify:

a  = -129    => false
cast (a as small int) =  small int(-129) => false
a  <= -129  => false
a  > -129    =>  true
a <= -128   =>  a = -128

currently data type check only support tinyint, small int, int, big int, decimalv3.

if data type is float like, compare them with literal may lost precision, but maybe suport it later.

This PR need more test.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yujun777
Copy link
Collaborator Author

run buildall

1 similar comment
@yujun777
Copy link
Collaborator Author

run buildall

morrySnow
morrySnow reviewed Nov 28, 2024
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java Outdated
}
}

public static Expression getTrue(Expression expression) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe trueOrNull is a better name?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe trueOrNull is a better name?

fix

fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java Outdated
Comment on lines 1818 to 1822
//minVal = BigDecimal.valueOf(-Float.MAX_VALUE);
return Optional.of(Pair.of(new BigDecimal(String.valueOf(Float.MIN_VALUE)),
new BigDecimal(String.valueOf(Float.MAX_VALUE))));
} else if (dataType.isDoubleType()) {
//minVal = BigDecimal.valueOf(-Double.MAX_VALUE);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need -Double.MAX_VALUE

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need -Double.MAX_VALUE

fix

fe/fe-core/src/main/java/org/apache/doris/nereids/util/TypeCoercionUtils.java Outdated
if (scale >= 0) {
StringBuilder sb = new StringBuilder();
for (int i = 0; i < precision - scale; i++) {
sb.append('9');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe u could use org.apache.common.lang3.StringUtils.repeat

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe u could use org.apache.common.lang3.StringUtils.repeat

fix

@yujun777 yujun777 force-pushed the simplify-comparison-with-limit branch from 67fd0d5 to 476a296 Compare November 29, 2024 01:40
@yujun777
Copy link
Collaborator Author

run buildall

@yujun777 yujun777 force-pushed the simplify-comparison-with-limit branch 3 times, most recently from c61cf89 to 5517819 Compare November 29, 2024 14:05
@yujun777
Copy link
Collaborator Author

run buildall

@yujun777 yujun777 force-pushed the simplify-comparison-with-limit branch from dd010e3 to e5ef895 Compare November 29, 2024 14:43
@yujun777
Copy link
Collaborator Author

run buildall

@yujun777 yujun777 force-pushed the simplify-comparison-with-limit branch 2 times, most recently from 4126bab to 7b19651 Compare November 30, 2024 05:23
@yujun777 yujun777 marked this pull request as draft December 2, 2024 01:28
@yujun777
Copy link
Collaborator Author

yujun777 commented Dec 2, 2024

prequire PR: #44831

@yujun777
Copy link
Collaborator Author

yujun777 commented Dec 3, 2024

run buildall

@yujun777 yujun777 marked this pull request as ready for review December 3, 2024 08:15
morrySnow
morrySnow reviewed Dec 3, 2024
View reviewed changes
regression-test/data/nereids_rules_p0/infer_predicate/pull_up_predicate_literal.out
--------PhysicalLimit[LOCAL]
----------PhysicalProject
------------PhysicalStorageLayerAggregate[test_pull_up_predicate_literal]
------PhysicalProject
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should modify pull up's case to ensure not generate PhysicalEmptyRelation

morrySnow
morrySnow reviewed Dec 3, 2024
View reviewed changes
regression-test/data/nereids_rules_p0/predicate_infer/infer_predicate.out
@@ -373,24 +373,13 @@ PhysicalResultSink

-- !infer9 --
PhysicalResultSink
--hashJoin[INNER_JOIN] hashCondition=((t1.id = t2.id)) otherCondition=()
----filter((cast(id as BIGINT) = 2147483648))
------PhysicalOlapScan[t1]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as pull up predicate cases

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as pull up predicate cases

update

@yujun777
Copy link
Collaborator Author

yujun777 commented Dec 3, 2024

run buildall

starocean999
starocean999 previously approved these changes Dec 6, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 6, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 6, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 6, 2024

PR approved by anyone and no changes requested.

englefly
englefly previously approved these changes Dec 9, 2024
View reviewed changes
@yujun777 yujun777 mentioned this pull request Dec 9, 2024
Merged
16 tasks
@yujun777 yujun777 force-pushed the simplify-comparison-with-limit branch from 828bc82 to a99911e Compare December 13, 2024 07:16
@yujun777 yujun777 force-pushed the simplify-comparison-with-limit branch from a99911e to 90382c9 Compare December 13, 2024 07:20
@yujun777 yujun777 force-pushed the simplify-comparison-with-limit branch from 90382c9 to 8201561 Compare December 13, 2024 07:20
@yujun777
Copy link
Collaborator Author

run buildall

@morrySnow morrySnow added the need more test Add more test label Dec 13, 2024
@morrySnow morrySnow merged commit 475571b into apache:master Dec 13, 2024
27 of 29 checks passed
starocean999 pushed a commit that referenced this pull request Dec 17, 2024
@yujun777
branch-3.0: [fix](nereids) fix compare with long min for simplify com…
0444e6c 
…parison rule (#44920)

pick the fix compare bug code from #44732. but #44732 not pick into 3.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow morrySnow approved these changes

@starocean999 starocean999 starocean999 approved these changes

@englefly englefly englefly left review comments

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. need more test Add more test reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@yujun777 @doris-robot @englefly @starocean999 @morrySnow