Using IBIS with Databricks

[nikhilmakan02]

Hi Team, just wondering if you can use Ibis with a Databricks as a backend given its Spark? I guess if not am I correct in saying given that you just pass in a SQL string to the Databricks SQL Connector in Python, you could use Ibis to compile the SQL? But just handle the connection and execute part yourself using the Databricks SQL Connector?

[gforsyth]

Hey @nikhilmakan02 -- sorry we missed this!

I haven't used databricks much, so I'm not entirely sure how to do this, but so long as you can get a pyspark session going, you can then use ibis to interact with it, something like:

>>> import ibis
>>> from pyspark.sql import SparkSession
>>> session = SparkSession.builder.getOrCreate()  # <-- thie line might be different on databricks?
>>> ibis.pyspark.connect(session)

[nikhilmakan02]

Thanks Gil, appreciate the response on this. I will see how I can incorporate that. Kind regards Nikhil Makan

…

On Thu, Feb 23, 2023 at 4:28 AM Gil Forsyth ***@***.***> wrote: Hey @nikhilmakan02 <https://github.com/nikhilmakan02> -- sorry we missed this! I haven't used databricks much, so I'm not entirely sure how to do this, but so long as you can get a pyspark session going, you can then use ibis to interact with it, something like: >>> import ibis>>> from pyspark.sql import SparkSession>>> session = SparkSession.builder.getOrCreate() # <-- thie line might be different on databricks?>>> ibis.pyspark.connect(session) — Reply to this email directly, view it on GitHub <#5510 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALQB2IUJ3ZHDZREYWWAIDIDWYYWAVANCNFSM6AAAAAAUXPTFJI> . You are receiving this because you were mentioned.Message ID: ***@***.***>