Trino Delta Lake connector fails to read _last_checkpoint file from MinIO

[MammadTavakoli]

I'm experiencing an issue with Trino's Delta Lake connector when trying to query a Delta Lake table stored in MinIO. My setup involves Trino, MinIO, and Hive Metastore, all running in Docker containers using a Docker Compose file.

Here are the key details about my setup:

1. I can successfully create a Delta Lake table in dw.test and save data to it.
2. When I connect to Trino via DBeaver and try to select from dw.test, I get the following error:

SQL Error [84279297]: Query failed (#20241203_091353_00046_vdmsn): Error getting snapshot for dw.test

The Trino logs show the following error:

2024-12-03T09:13:58.130ZDEBUGQuery-20241203_091353_00046_vdmsn-438io.trino.plugin.deltalake.transactionlog.TransactionLogParserFailure when accessing last checkpoint informa
tion, will be retried
java.lang.IllegalArgumentException: No factory for location: s3a://warehouse/dw.db/test/_delta_log/_last_checkpoint
at io.trino.filesystem.manager.FileSystemModule.lambda$createFileSystemFactory$2(FileSystemModule.java:149)
at java.base/java.util.Optional.orElseThrow(Optional.java:403)
at io.trino.filesystem.manager.FileSystemModule.lambda$createFileSystemFactory$3(FileSystemModule.java:149)
at io.trino.filesystem.switching.SwitchingFileSystem.fileSystem(SwitchingFileSystem.java:194)
at io.trino.filesystem.switching.SwitchingFileSystem.newInputFile(SwitchingFileSystem.java:60)
at io.trino.filesystem.tracing.TracingFileSystem.newInputFile(TracingFileSystem.java:51)
at io.trino.plugin.deltalake.transactionlog.TransactionLogParser.tryReadLastCheckpoint(TransactionLogParser.java:267)
at io.trino.plugin.deltalake.transactionlog.TransactionLogParser.lambda$readLastCheckpoint$2(TransactionLogParser.java:260)
at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
at io.trino.plugin.deltalake.transactionlog.TransactionLogParser.readLastCheckpoint(TransactionLogParser.java:260)
at io.trino.plugin.deltalake.transactionlog.TransactionLogAccess.loadSnapshot(TransactionLogAccess.java:176)
at io.trino.plugin.deltalake.DeltaLakeMetadata.getSnapshot(DeltaLakeMetadata.java:518)
at io.trino.plugin.deltalake.DeltaLakeMetadata.getTableHandle(DeltaLakeMetadata.java:628)
at io.trino.plugin.deltalake.DeltaLakeMetadata.getTableHandle(DeltaLakeMetadata.java:355)
at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorMetadata.getTableHandle(ClassLoaderSafeConnectorMetadata.java:1237)
at io.trino.tracing.TracingConnectorMetadata.getTableHandle(TracingConnectorMetadata.java:142)
at io.trino.metadata.MetadataManager.lambda$getTableHandle$5(MetadataManager.java:292)
at java.base/java.util.Optional.flatMap(Optional.java:289)
at io.trino.metadata.MetadataManager.getTableHandle(MetadataManager.java:283)
at io.trino.metadata.MetadataManager.getRedirectionAwareTableHandle(MetadataManager.java:1968)
at io.trino.metadata.MetadataManager.getRedirectionAwareTableHandle(MetadataManager.java:1960)
at io.trino.tracing.TracingMetadata.getRedirectionAwareTableHandle(TracingMetadata.java:1494)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.getTableHandle(StatementAnalyzer.java:5844)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitTable(StatementAnalyzer.java:2293)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitTable(StatementAnalyzer.java:522)
at io.trino.sql.tree.Table.accept(Table.java:60)
at io.trino.sql.tree.AstVisitor.process(AstVisitor.java:27)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:541)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.analyzeFrom(StatementAnalyzer.java:4893)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitQuerySpecification(StatementAnalyzer.java:3093)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitQuerySpecification(StatementAnalyzer.java:522)
at io.trino.sql.tree.QuerySpecification.accept(QuerySpecification.java:155)
at io.trino.sql.tree.AstVisitor.process(AstVisitor.java:27)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:541)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:549)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitQuery(StatementAnalyzer.java:1564)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitQuery(StatementAnalyzer.java:522)
at io.trino.sql.tree.Query.accept(Query.java:119)
at io.trino.sql.tree.AstVisitor.process(AstVisitor.java:27)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:541)
at io.trino.sql.analyzer.StatementAnalyzer.analyze(StatementAnalyzer.java:501)
at io.trino.sql.analyzer.StatementAnalyzer.analyze(StatementAnalyzer.java:490)
at io.trino.sql.analyzer.Analyzer.analyze(Analyzer.java:98)
at io.trino.sql.analyzer.Analyzer.analyze(Analyzer.java:87)
at io.trino.execution.SqlQueryExecution.analyze(SqlQueryExecution.java:289)
at io.trino.execution.SqlQueryExecution.<init>(SqlQueryExecution.java:222)
at io.trino.execution.SqlQueryExecution$SqlQueryExecutionFactory.createQueryExecution(SqlQueryExecution.java:892)
at io.trino.dispatcher.LocalDispatchQueryFactory.lambda$createDispatchQuery$0(LocalDispatchQueryFactory.java:153)
at io.trino.$gen.Trino_464____20241203_071415_2.call(Unknown Source)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1575)


2024-12-03T09:13:58.131ZDEBUGdispatcher-query-33io.trino.execution.QueryStateMachineQuery 20241203_091353_00046_vdmsn is FAILED
2024-12-03T09:13:58.131ZDEBUGdispatcher-query-35io.trino.execution.QueryStateMachineQuery 20241203_091353_00046_vdmsn failed
io.trino.spi.TrinoException: Error getting snapshot for dw.test
at io.trino.plugin.deltalake.DeltaLakeMetadata.getSnapshot(DeltaLakeMetadata.java:525)
at io.trino.plugin.deltalake.DeltaLakeMetadata.getTableHandle(DeltaLakeMetadata.java:628)
at io.trino.plugin.deltalake.DeltaLakeMetadata.getTableHandle(DeltaLakeMetadata.java:355)
at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorMetadata.getTableHandle(ClassLoaderSafeConnectorMetadata.java:1237)
at io.trino.tracing.TracingConnectorMetadata.getTableHandle(TracingConnectorMetadata.java:142)
at io.trino.metadata.MetadataManager.lambda$getTableHandle$5(MetadataManager.java:292)
at java.base/java.util.Optional.flatMap(Optional.java:289)
at io.trino.metadata.MetadataManager.getTableHandle(MetadataManager.java:283)
at io.trino.metadata.MetadataManager.getRedirectionAwareTableHandle(MetadataManager.java:1968)
at io.trino.metadata.MetadataManager.getRedirectionAwareTableHandle(MetadataManager.java:1960)
at io.trino.tracing.TracingMetadata.getRedirectionAwareTableHandle(TracingMetadata.java:1494)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.getTableHandle(StatementAnalyzer.java:5844)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitTable(StatementAnalyzer.java:2293)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitTable(StatementAnalyzer.java:522)
at io.trino.sql.tree.Table.accept(Table.java:60)
at io.trino.sql.tree.AstVisitor.process(AstVisitor.java:27)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:541)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.analyzeFrom(StatementAnalyzer.java:4893)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitQuerySpecification(StatementAnalyzer.java:3093)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitQuerySpecification(StatementAnalyzer.java:522)
at io.trino.sql.tree.QuerySpecification.accept(QuerySpecification.java:155)
at io.trino.sql.tree.AstVisitor.process(AstVisitor.java:27)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:541)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:549)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitQuery(StatementAnalyzer.java:1564)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitQuery(StatementAnalyzer.java:522)
at io.trino.sql.tree.Query.accept(Query.java:119)
at io.trino.sql.tree.AstVisitor.process(AstVisitor.java:27)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:541)
at io.trino.sql.analyzer.StatementAnalyzer.analyze(StatementAnalyzer.java:501)
at io.trino.sql.analyzer.StatementAnalyzer.analyze(StatementAnalyzer.java:490)
at io.trino.sql.analyzer.Analyzer.analyze(Analyzer.java:98)
at io.trino.sql.analyzer.Analyzer.analyze(Analyzer.java:87)
at io.trino.execution.SqlQueryExecution.analyze(SqlQueryExecution.java:289)
at io.trino.execution.SqlQueryExecution.<init>(SqlQueryExecution.java:222)
at io.trino.execution.SqlQueryExecution$SqlQueryExecutionFactory.createQueryExecution(SqlQueryExecution.java:892)
at io.trino.dispatcher.LocalDispatchQueryFactory.lambda$createDispatchQuery$0(LocalDispatchQueryFactory.java:153)
at io.trino.$gen.Trino_464____20241203_071415_2.call(Unknown Source)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1575)
Caused by: java.lang.IllegalArgumentException: No factory for location: s3a://warehouse/dw.db/test/_delta_log/_last_checkpoint
at io.trino.filesystem.manager.FileSystemModule.lambda$createFileSystemFactory$2(FileSystemModule.java:149)
at java.base/java.util.Optional.orElseThrow(Optional.java:403)
at io.trino.filesystem.manager.FileSystemModule.lambda$createFileSystemFactory$3(FileSystemModule.java:149)
at io.trino.filesystem.switching.SwitchingFileSystem.fileSystem(SwitchingFileSystem.java:194)
at io.trino.filesystem.switching.SwitchingFileSystem.newInputFile(SwitchingFileSystem.java:60)
at io.trino.filesystem.tracing.TracingFileSystem.newInputFile(TracingFileSystem.java:51)
at io.trino.plugin.deltalake.transactionlog.TransactionLogParser.tryReadLastCheckpoint(TransactionLogParser.java:267)
at io.trino.plugin.deltalake.transactionlog.TransactionLogParser.lambda$readLastCheckpoint$2(TransactionLogParser.java:260)
at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
at io.trino.plugin.deltalake.transactionlog.TransactionLogParser.readLastCheckpoint(TransactionLogParser.java:260)
at io.trino.plugin.deltalake.transactionlog.TransactionLogAccess.loadSnapshot(TransactionLogAccess.java:176)
at io.trino.plugin.deltalake.DeltaLakeMetadata.getSnapshot(DeltaLakeMetadata.java:518)
... 43 more


2024-12-03T09:13:58.132ZINFOdispatcher-query-33io.trino.event.QueryMonitorTIMELINE: Query 20241203_091353_00046_vdmsn :: FAILED (DELTA_LAKE_INVALID_SCHEMA) :: elapsed 5
048ms :: planning 5048ms :: waiting 0ms :: scheduling 0ms :: running 0ms :: finishing 0ms :: begin 2024-12-03T09:13:53.082Z :: end 2024-12-03T09:13:58.130Z
expalnation:

that i think it is:

Caused by: java.lang.IllegalArgumentException: No factory for location: s3a://warehouse/dw.db/test/_delta_log/_last_checkpoint

My Trino configuration in the Docker Compose file looks like this:

trino-coordinator:
  image: "trinodb/trino:latest"
  container_name: trino-coordinator
  ports:
    - 8080:8080  
  volumes:
    - ./trino/coordinator/etc/catalog:/etc/trino/catalog 
    - ./trino/coordinator/etc:/etc/trino  
  environment:
    - TRINO_CATALOG_PATH=/etc/trino/catalog  
  depends_on:
    - hive-metastore 
  networks:
    - dl-network
  deploy:
    resources:
      limits:
        cpus: '1'
        memory: 10G

My Delta Lake catalog configuration includes:

connector.name=delta_lake

# Hive Metastore details
hive.metastore.uri=thrift://hive-metastore:9083 

# MinIO details
s3.endpoint=http://minio:9000 
s3.aws-access-key=test 
s3.aws-secret-key=test12334567 
s3.path-style-access=true

I've also configured my Hive Metastore with the following environment variables:

environment:
  HIVE_METASTORE_DRIVER: org.postgresql.Driver
      
  HIVE_METASTORE_JDBC_URL: jdbc:postgresql://metastore_db:5432/metastore
  
  HIVE_METASTORE_USER: hive
  HIVE_METASTORE_PASSWORD: hive
  
  HIVE_METASTORE_WAREHOUSE_DIR: s3://warehouse/
  
  S3_ENDPOINT: http://minio:9000
  
  S3_ACCESS_KEY: test
  S3_SECRET_KEY: test12334567
  
  S3_PATH_STYLE_ACCESS: "true"

I've tried various queries, including SELECT * FROM dw.test, but I consistently get the error about failing to get the snapshot.

Could this be related to permissions issues with MinIO access? Or could it be a known issue with Trino and Delta Lake connectors?

Question

How can I troubleshoot this issue with Trino failing to read the _last_checkpoint file from MinIO? Are there any known issues or common solutions to this problem?

But this code for reading from that location run correctly:

from pyspark.sql import SparkSession
from delta.tables import DeltaTable


def get_spark_session():
    return (
        SparkSession.builder.appName("DeltaLakeApp")
        .config("spark.jars.packages", "io.delta:delta-core_2.12:3.2.1")
        .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
        .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
        .config("spark.hadoop.fs.s3a.endpoint", "http://minio:9000")\
        .config("spark.hadoop.fs.s3a.access.key", "test")\
        .config("spark.hadoop.fs.s3a.secret.key", "test12334567")\
        .getOrCreate()
    )

def main():     
    spark = get_spark_session()
     
    table_path = "s3a://warehouse/dw.db/test"
    df = DeltaTable.forPath(spark, table_path).toDF()
    df.show()
         
     
def run_job(spark, config):
     main()

Thank you for your help!