Support multi string contains [databricks] #11413
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
res-life
force-pushed
the
multi-string-contians
branch
from
27654c6
to
4444aa4
Compare
Signed-off-by: Chong Gao <[email protected]>
res-life
force-pushed
the
multi-string-contians
branch
from
4444aa4
to
3677ff6
Compare
res-life
changed the title
|
Building failed, because it's depending on rapidsai/cudf#16641 |
res-life
changed the title
revans2
previously approved these changes
Sep 4, 2024
gerashegalov
previously approved these changes
Sep 7, 2024
| @@ -379,3 +379,30 @@ def test_case_when_all_then_values_are_scalars_with_nulls(): | |||
| "tab", | |||
| sql_without_else, | |||
| conf = {'spark.rapids.sql.case_when.fuse': 'true'}) | |||
|
|
|||
| @pytest.mark.parametrize('combine_string_contains_enabled', ['true', 'false']) | |||
There was a problem hiding this comment.
nit: prefer Python constants
Suggested change
| @pytest.mark.parametrize('combine_string_contains_enabled', ['true', 'false']) | |
| @pytest.mark.parametrize('combine_string_contains_enabled', [True, False]) |
However, the pytest case id will be more readable if, instead of a boolean, parameters are strings
@pytest.mark.parametrize('string_contains_mode', ['multiContains', 'singleContains'], ids=idfn)
Comment on lines
+471
to
+473
| override def equals(o: Any): Boolean = o match { | ||
| case other: ContainsCombiner => exp.left.semanticEquals(other.exp.left) && | ||
| exp.right.isInstanceOf[GpuLiteral] && other.exp.right.isInstanceOf[GpuLiteral] |
There was a problem hiding this comment.
nit: if you make ContainsCombiner a case class you can use pattern matching instead of manual instanceof checks:
Suggested change
| override def equals(o: Any): Boolean = o match { | |
| case other: ContainsCombiner => exp.left.semanticEquals(other.exp.left) && | |
| exp.right.isInstanceOf[GpuLiteral] && other.exp.right.isInstanceOf[GpuLiteral] | |
| override def equals(o: Any): Boolean = (o, exp) match { | |
| case (ContainsCombiner(GpuContains(combLeft, GpuLiteral(_, _))), GpuContains(expLeft, GpuLiteral(_, _))) => | |
| expLeft.semanticEquals(combLeft) |
ttnghia
changed the title
res-life
dismissed stale reviews from gerashegalov and revans2
The base branch was changed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Combine multi targets for multiple string contains, and invoke one kernel to get multiple bool columns results.
Depends on cuDF PR
Verified:
containsincase whenwere combined when running pytest.Perf test
summary
details
short strings
The above part is kernel time of new kernel
The bottom part is base line.
long strings
Kernel time:
Base line:
new:
