Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature fix load as spark #509

Closed
wants to merge 3 commits into from
Closed

Feature fix load as spark #509

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
22f3124
Fixed issues in load_as_spark
Shabbir-Khan-12 Jun 20, 2024
a504feb
Revert "Fixed issues in load_as_spark"
Shabbir-Khan-12 Jun 20, 2024
4f5324e
Fixed issues in load_as_spark
Shabbir-Khan-12 Jun 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 1 addition & 5 deletions python/delta_sharing/delta_sharing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,11 +150,7 @@ def load_as_spark(
except ImportError:
raise ImportError("Unable to import pyspark. `load_as_spark` requires PySpark.")

spark = SparkSession.getActiveSession()
assert spark is not None, (
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we still want to keep the assert?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will not be beneficial to keep it. If any case occurs with issues in spark session fetching then this assert will handle that.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to hear your thoughts as to what should be the better approach.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay, I'm reading this, it seems it will default to None, so I think this assert is still helpful. https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/SparkSession.html

"No active SparkSession was found. "
"`load_as_spark` requires running in a PySpark application."
)
spark = SparkSession.active()
df = spark.read.format("deltaSharing")
if version is not None:
df.option("versionAsOf", version)
Expand Down
Loading