SNOW-977836 Update dataframe.py #1149
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
the upper() function is converting the column name to upper case and hence when the dataframe has two columns with same name but different case as show Ex- Column names = ['Snow Flake', 'SNOW FLAKE']. When the user tries to rename 'Snow Flake' column to 'Snow Flake Renamed', the current withColumnRenamed method throws an exception as the method converts 'Snow Flake' to upper case but since the dataframe already has another column called 'SNOW FLAKE', the to_be_renamed list will have 2 elements and hence an exception will be raised.
|
CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅ |
|
I have read the CLA Document and I hereby sign the CLA |
Ilyas-kipi
changed the title
|
certainly fixes the issue, but I think tests need to be added in Also please see my comment on renaming of duplicated columns. |
|
Sure, will add test cases around this functionality |
Added test case for with_column_renamed function when dealing with columns having same name but in a different case style
|
Hey @suenalaba , pls see my comments on #1148 , I've added test case to test_dataframe.py with the name test_with_column_renamed_case_sensitivity() |
|
looks good to me now |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refer this doc -> https://docs.google.com/document/d/1BtcercvMKIqaMUzLWMDrJqS_JlKTniMFKqxGxB5FxiA
In the withColumnRenamed function, the string function upper() is converting the column name to upper case and hence when the dataframe has two columns with same name but different case as show Ex- Column names = ['Snow Flake', 'SNOW FLAKE']. When the user tries to rename 'Snow Flake' column to 'Snow Flake Renamed', the current withColumnRenamed method throws an exception as the method converts 'Snow Flake' to upper case but since the dataframe already has another column called 'SNOW FLAKE', the to_be_renamed list will have 2 elements and hence an exception will be raised.
Please answer these questions before submitting your pull requests. Thanks!
What GitHub issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes - SNOW-977836: The withColumnRenamed fucntion fails to rename a column if the snowpark dataframe has multiple columns with same name but with different case style #1148
Fill out the following pre-review checklist:
Please describe how your code solves the related issue.
In the withColumnRenamed function in dataframe.py file, in line 3541, upper() function is converting the column name to upper case and hence when the dataframe has two columns with same name but different case as show Ex- Column names = ['Snow Flake', 'SNOW FLAKE']. When the user tries to rename 'Snow Flake' column to a name called 'Snow Flake Renamed', the current withColumnRenamed method throws an exception as the method converts 'Snow Flake' to upper case but since the dataframe already has another column called 'SNOW FLAKE', the to_be_renamed list will have 2 elements and hence an exception will be raised. Removing the upper function will make sure that we compare the columns to be renamed and the existing dataframe columns in the same case as they exist and will not raise an exception if we have multiple column with same name but different case.