kedro-datasets: ExternalTableDataset for Databricks

[MinuraPunchihewa]

Description

I use Kedro on Databricks heavily, but in most of my projects the datasets in use are External tables (for a myriad of reasons) and I am currently unable to perform my IO operations easily.

Context

As I have said above, my datasets are maintained as External tables and I am certain there are many others users (and organizations) out there that follow a similar pattern, at least for some of their datasets.

Possible Implementation

It should be possible to implement at ExternalTableDataset for Kedro to allow the said IO operations.

Possible Alternatives

I have considered converting all of my datasets on Databricks to Managed tables, but at the moment, this does not make sense for me from an architectural point of view.

[MinuraPunchihewa]

Hey @astrojuanlu and the Kedro team,
I would love to work on this and make a contribution if you believe it would be useful.

[astrojuanlu]

Hi @MinuraPunchihewa , if you want to open a pull request please go ahead!

[MinuraPunchihewa]

Thanks, @astrojuanlu. Will do.