fix: Incorrect .join(..., how="left").head(N) if N <= left_df.height() and there are duplicate matches
#19422
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nameexhaustion
changed the title
.join(..., how="left").head(N) if N <= left_df.height().join(..., how="left").head(N) if N <= left_df.height() and there are duplicate matches
github-actions
bot
added
fix
nameexhaustion
commented
Oct 24, 2024
| @@ -104,6 +104,7 @@ all = [ | |||
| "parquet", | |||
| "ipc", | |||
| "polars-python/all", | |||
| "performant", | |||
There was a problem hiding this comment.
Code-path for left join on sorted keys is also affected, but it is gated behind the performant feature flag so it doesn't get compiled in debug builds.
I think maybe instead of adding it here, we can add a test step to the Python release workflow that runs through the test suite with the optimized release build?
There was a problem hiding this comment.
I assumed it was always included in pypolars. It should always be activated.
nameexhaustion
requested review from
ritchie46,
c-peters,
alexander-beedie,
MarcoGorelli,
reswqa and
orlp
as code owners
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #19422 +/- ##
=======================================
Coverage 80.03% 80.04%
=======================================
Files 1532 1532
Lines 210749 210766 +17
Branches 2442 2442
=======================================
+ Hits 168676 168701 +25
+ Misses 41518 41510 -8
Partials 555 555 ☔ View full report in Codecov by Sentry. |
|
Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #19405
Fixes #19403
A long time ago a fast-path was added to left-join that would identify a left-join didn't have any duplicate matches by checking the height of the join_tuples matches the height of the input DF:
2f5f3ec#diff-33db4b397429a6dc2297400e2fe812fb12f17ed977dfeee5f96e34fdb9b05757R728-R730
At some point we added slicing to the input left DF and also the join tuples, meaning that we would incorrectly identify the left-join as having no duplicate matches.