Skip to content

This library allows Scala and Java-based projects (including Apache Flink, Apache Hive, Apache Beam, and PrestoDB) to read from and write to Delta Lake.

License

Notifications You must be signed in to change notification settings

rahulsmahadev/connectors

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Delta Lake Logo Connectors

Test License

We are building connectors to bring Delta Lake to popular big-data engines outside Apache Spark (e.g., Apache Hive, Presto, Apache Flink) and also to common reporting tools like Microsoft Power BI.

Introduction

This is the repository for Delta Lake Connectors. It includes

Please refer to the main Delta Lake repository if you want to learn more about the Delta Lake project.

API documentation

Delta Standalone

Delta Standalone, formerly known as the Delta Standalone Reader (DSR), is a JVM library to read and write Delta tables. Unlike https://github.com/delta-io/delta, this project doesn't use Spark to read or write tables and it has only a few transitive dependencies. It can be used by any application that cannot use a Spark cluster.

  • To compile the project, run build/sbt standalone/compile
  • To test the project, run build/sbt standalone/test
  • To publish the JAR, run build/sbt standaloneCosmetic/publishM2

See Delta Standalone for detailed documentation.

Connectors

Hive Connector

Read Delta tables directly from Apache Hive using the Hive Connector. See the dedicated README.md for more details.

Flink/Delta Connector

Use the Flink/Delta Connector to read and write Delta tables from Apache Flink applications. The connector includes a sink for writing to Delta tables from Apache Flink, and a source for reading Delta tables using Apache Flink (still in progress.) See the dedicated README.md for more details.

sql-delta-import

sql-delta-import allows for importing data from a JDBC source into a Delta table.

Power BI connector

The connector for Microsoft Power BI is basically just a custom Power Query function that allows you to read a Delta table from any file-based data source supported by Microsoft Power BI. Details can be found in the dedicated README.md.

Reporting issues

We use GitHub Issues to track community reported issues. You can also contact the community for getting answers.

Contributing

We welcome contributions to Delta Lake Connectors repository. We use GitHub Pull Requests for accepting changes.

Community

There are two mediums of communication within the Delta Lake community.

Local Development & Testing

  • Before local debugging of standalone tests in IntelliJ, run all standalone tests using SBT. This helps IntelliJ recognize the golden tables as class resources.

About

This library allows Scala and Java-based projects (including Apache Flink, Apache Hive, Apache Beam, and PrestoDB) to read from and write to Delta Lake.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Java 58.8%
  • Scala 40.6%
  • Shell 0.6%