beeline query from s3 through kyuubi failed.

[PandyYang]

condition

By connecting to the 10000 port of hiveserver2, the query can be performed normally, but not through kyuubi. Do I need to introduce the aws package under kyuubi's lib? At present, AWS dependencies have been introduced in hive spark hadoop related directories.

error

2023-12-20 09:35:56,600 INFO scheduler.DAGScheduler: Job 1 failed: collect at ExecuteStatement.scala:90, took 2.759964 s
2023-12-20 09:35:56,612 INFO kyuubi.SQLOperationListener: Finished stage: Stage(1, 0); Name: 'collect at ExecuteStatement.scala:90'; Status: failed(Job aborted due to stage failure: Task 1 in stage 1.0 failed 4 times, most recent failure: Lost task 1.3 in stage 1.0 (TID 5) (hadoop102 executor 1): java.lang.NoSuchMethodError: com.amazonaws.http.HttpResponse.getHttpRequest()Lcom/amazonaws/thirdparty/apache/http/client/methods/HttpRequestBase;
        at com.amazonaws.services.s3.internal.S3ObjectResponseHandler.abortableIs(S3ObjectResponseHandler.java:64)
        at com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:57)
        at com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:29)
        at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:69)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1627)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1336)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272)
        at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1409)
        at org.apache.hadoop.fs.s3a.S3AInputStream.lambda$reopen$0(S3AInputStream.java:227)
        at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:115)
        at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:226)
        at org.apache.hadoop.fs.s3a.S3AInputStream.lambda$lazySeek$1(S3AInputStream.java:392)
        at org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$3(Invoker.java:228)
        at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:115)
        at org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$5(Invoker.java:352)
        at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:412)
        at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:348)
        at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:226)
        at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:270)
        at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:384)
        at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:503)
        at java.io.DataInputStream.read(DataInputStream.java:149)
        at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.fillBuffer(UncompressedSplitLineReader.java:62)
        at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:227)
        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:185)
        at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.readLine(UncompressedSplitLineReader.java:94)
        at org.apache.hadoop.mapred.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:221)
        at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:259)
        at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:50)
        at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:312)
        at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:243)
        at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
        at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
        at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:349)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
        at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:131)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

[pan3793]

The error stacktrace indicates that you have classpath issue on your Spark driver

[pan3793]

I suppose the same error will happen if you run the same query through $SPARK_HOME/bin/spark-sql. You may need to tune jars under $SPARK_HOME/jars

[PandyYang]

hi. thanks for your reply. spark sql can be executed normally。
scala> spark.sql("show tables").show
2023-12-20 10:08:34,961 WARN conf.HiveConf: HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
2023-12-20 10:08:35,336 WARN conf.HiveConf: HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
2023-12-20 10:08:35,336 WARN conf.HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
2023-12-20 10:08:35,337 WARN conf.HiveConf: HiveConf of name hive.stats.retries.wait does not exist
2023-12-20 10:08:36,575 WARN conf.HiveConf: HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
2023-12-20 10:08:37,801 WARN metastore.ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
+---------+----------+-----------+
|namespace| tableName|isTemporary|
+---------+----------+-----------+
| default| mgtest2| false|
| default|test_mysql| false|
+---------+----------+-----------+

scala> spark.sql("select * from mgtest2").show
+---+
| id|
+---+
| 1|
| 2|
+---+

[pan3793]

Check the following things:

• spark-sql always launch Spark application on the client mode, but Kyuubi would respect the spark.submit.deployMode, does client/cluster mode have same issues?
• Have you put additional artifacts under the classpath of the Kyuubi bootstrapped Spark application? e.g. add jar for UDF
• check the Environment tab on the Spark UI to find diff

[PandyYang]

hi~ I solved this problem. according to your tips, finally, I found that the versions used by hadoop and spark did not match, which caused compatibility issues in the aws jar package. I have changed the cluster and hadoop version. Hope you have a nice day, thank you very much.
now the version i used is.
hadoop 3.3.1
spark 3.2.1
hive 3.1.2
kyuubi 1.5.2
hadoop-aws 3.3.1
aws-java-sdk-bundle 1.11.901