Lance Java Readers

[wmoustafa]

Are there native Java readers for Lance files (e.g., readers that would load Lance files to Arrow objects purely in the JVM)? Looks like there is not from my preliminary research. If they do not exist, what is the best practice to read Lance files from JVM-based engines (e.g., Spark)?

[wmoustafa]

To clarify, the objective is to implement a Spark DataSource that scans Lance files, i.e., not necessarily use Spark out-of-the-box and read through intermediate Pandas dataframes -- the latter is not the objective of this discussion.

[LuQQiu]

Hi @wmoustafa, we have a Java reader for Lance files under https://github.com/lancedb/lance/tree/main/java
We also have some initial implementation of Spark connector under https://github.com/lancedb/lance/tree/main/java/spark
The java spark connector still require some development work
We are also cooperating with @allisonwang-db on the pyspark data source allisonwang-db