DBeaver: How to get tables metadata

[mmuru]

I am using Kyuubi 1.3.0 release with DBeaver 21.1.5. In DBeaver, I could see tables listed in spark_catalog.default schema but it does not show tables metadata (e.g columns and datatypes) and empty. However, I could run describe table table-name and returns columns/datatypes. Please, let me know if I missed any steps or configurations? Thanks.

[yaooqinn]

can you provide the server and engine logs?

[mmuru]

@yaooqinn: Sure, attached log file
kyuubi-muru-org.apache.kyuubi.server.KyuubiServer-Murus-MacBook-Pro.local.out.zip

[pan3793]

It should be fixed in #1133, would you please try the latest master/branch-1.3?

[mmuru]

@pan3793: I tried the latest master and still the same result. It didn't show columns.

[pan3793]

Both DBeaver bundled Hive JDBC Driver and Kyuubi provided Hive JDBC driver work well on my laptop.

The difference between these two is that vanilla Hive JDBC driver doesn't support multi catalogs, Kyuubi provided one fixed this issue, you are not in such a case, I think both two should work well.

Using DBeaver bundled Hive JDBC Driver to connect Kyuubi Server

To use Kyuubi provided Hive JDBC driver, please first checkout code from the master branch, then run build/mvn clean install to build the latest Kyuubi JDBC jars.

[mmuru]

@pan3793: I tried Kyuubi hive JDBC driver as per your instructions but still the same issue. It still did not show the columns. I used spark engine version 3.0.3 (as spark-provided). Please, can you provide the engine version information?

Attached both Kyuubi and spark-engine log files
kyuubi-spark-sql-engine.log.0.zip
kyuubi-muru-org.apache.kyuubi.server.KyuubiServer-Murus-MacBook-Pro.local.out.zip

.

[pan3793]

Tested with Spark 3.1.2, let me try 3.0.3.

[mmuru]

FYI: I tried Spark 3.1.2 too but the result was the same. Using the Spark bundled Hadoop 3.2.0 and Hive 2.3.7. I am using Postgresql as Hive metastore. Please, let me know if you need more information.

[yaooqinn]

It works for me on my laptop

[pan3793]

Previous answers based on Hive tables and Iceberg tables, let me try Delta tables

[pan3793]

@mmuru I can reproduce with delta table, will investigate it later

[mmuru]

@pan3793: Thanks for the update and looking into it. Please, let me know if you want me to create issue for this.

[yaooqinn]

Hi @mmuru, sounds good to me to create an issue to track this.

@pan3793: Thanks for the update and looking into it. Please, let me know if you want me to create issue for this.

[pan3793]

After some investigation, I found it's a known issue of delta delta-io/delta#573

[pan3793]

Based on delta v1.0.0 source code, delta will erase schema before persisting table information into Hive metastore. The behavior doesn't change in master branch.

https://github.com/delta-io/delta/blob/5571d95966e3b03c4699dcfa1a5cd86bb16447dc/core/src/main/scala/org/apache/spark/sql/delta/commands/CreateDeltaTableCommand.scala#L387-L405

DBevaer uses GET_COLUMNS operation to ask Kyuubi for table column information, and Kyuubi handles it by calling Spark Catalog API, which retrieves metadata from Hive metastore, then you got nothing.

[mmuru]

@pan3793: Thanks for the update. Please, can you point me to where in Kyuubi side calling Spark catalog API? Is it something Kyuubi
side can handle it?

[pan3793]

where in Kyuubi side calling Spark catalog API?

https://github.com/apache/incubator-kyuubi/blob/ddfd0355e8867c7ea51f511a0c0023c3bf0e7e6e/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/shim/CatalogShim_v2_4.scala#L91-L104

Is it something Kyuubi side can handle it?

Technically, Kyuubi can, but not reasonable, it's a delta issue.

Kyuubi can use spark.table(xxx).schema to retrieve table schema to suppress this issue, but it makes GET_COLUMNS be a heavy operation rather than a lightweight metadata operation.

[mmuru]

@pan3793: Thanks for the details. I understand it is delta side issue. In the meantime, we could fix it in Kyuubi side till it is addressed in delta side.