kyuubi initializing error #5681

jseungtaek · 2023-11-13T09:38:27Z

jseungtaek
Nov 13, 2023

Hello, I'd like to use kyuubi as a spark and interactive. I applied kyuubi as pod in eks environment. The following message occurs when you access the eks node address through the beeline connection.

We created eks' service as NodePort and set it to access 30009.

run the ec2 instance as bin/kyubi start and access it, it runs normally, but I don't know why it doesn't work with eks.

I don't know anything about kyuubi, whether it's possible to use the spark cluster pod or how to write the image.

I'd really appreciate your help

use
image : kyuubi-1.8.0-spark

Warn: Not find kyuubi environment file /home/ubuntu/apache-kyuubi-1.8.0-bin/conf/kyuubi-env.sh, using default ones...
Connecting to jdbc:hive2://ip-172-31-17-232.ap-northeast-2.compute.internal:30009
2023-11-13 09:24:31.930 INFO KyuubiSessionManager-exec-pool: Thread-89 org.apache.kyuubi.operation.LaunchEngine: Processing apache's query[8708a9f9-7726-4de9-af6c-daed71ac6593]: PENDING_STATE -> RUNNING_STATE, statement:
LaunchEngine
2023-11-13 09:24:31.931 INFO KyuubiSessionManager-exec-pool: Thread-89 org.apache.kyuubi.shaded.curator.framework.imps.CuratorFrameworkImpl: Starting
2023-11-13 09:24:31.931 INFO KyuubiSessionManager-exec-pool: Thread-89 org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Initiating client connection, connectString=172.31.28.171:2181 sessionTimeout=60000 watcher=org.apache.kyuubi.shaded.curator.ConnectionState@58fb0b97
2023-11-13 09:24:31.932 INFO KyuubiSessionManager-exec-pool: Thread-89-SendThread(kyuubi-deployment-example-5f794459b-mzkqn:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Opening socket connection to server kyuubi-deployment-example-5f794459b-mzkqn/172.31.28.171:2181. Will not attempt to authenticate using SASL (unknown error)
2023-11-13 09:24:31.933 INFO KyuubiSessionManager-exec-pool: Thread-89-SendThread(kyuubi-deployment-example-5f794459b-mzkqn:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Socket connection established to kyuubi-deployment-example-5f794459b-mzkqn/172.31.28.171:2181, initiating session
2023-11-13 09:24:31.935 INFO KyuubiSessionManager-exec-pool: Thread-89-SendThread(kyuubi-deployment-example-5f794459b-mzkqn:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Session establishment complete on server kyuubi-deployment-example-5f794459b-mzkqn/172.31.28.171:2181, sessionid = 0x100003e18160004, negotiated timeout = 60000
2023-11-13 09:24:31.935 INFO KyuubiSessionManager-exec-pool: Thread-89-EventThread org.apache.kyuubi.shaded.curator.framework.state.ConnectionStateManager: State change: CONNECTED
2023-11-13 09:26:12.001 INFO KyuubiSessionManager-exec-pool: Thread-89 org.apache.kyuubi.engine.ProcBuilder: Logging to /opt/kyuubi/work/apache/kyuubi-spark-sql-engine.log.2
2023-11-13 09:26:12.002 INFO KyuubiSessionManager-exec-pool: Thread-89 org.apache.kyuubi.engine.EngineRef: Launching engine:
/opt/kyuubi/externals/spark-3.4.1-bin-hadoop3/bin/spark-submit \
	--class org.apache.kyuubi.engine.spark.SparkSQLEngine \
	--conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
	--conf spark.kyuubi.client.ipAddress=172.31.6.61 \
	--conf spark.kyuubi.client.version=1.8.0 \
	--conf spark.kyuubi.engine.engineLog.path=/opt/kyuubi/work/apache/kyuubi-spark-sql-engine.log.2 \
	--conf spark.kyuubi.engine.submit.time=1699867571998 \
	--conf spark.kyuubi.ha.addresses=172.31.28.171:2181 \
	--conf spark.kyuubi.ha.engine.ref.id=6724f97a-41b7-4a29-9313-71cd7b7295d5 \
	--conf spark.kyuubi.ha.namespace=/kyuubi_1.8.0_USER_SPARK_SQL/apache/default \
	--conf spark.kyuubi.ha.zookeeper.auth.type=NONE \
	--conf spark.kyuubi.server.ipAddress=0.0.0.0 \
	--conf spark.kyuubi.session.connection.url=0.0.0.0:10009 \
	--conf spark.kyuubi.session.real.user=apache \
	--conf spark.app.name=kyuubi_USER_SPARK_SQL_apache_default_6724f97a-41b7-4a29-9313-71cd7b7295d5 \
	--conf spark.kubernetes.driver.label.kyuubi-unique-tag=6724f97a-41b7-4a29-9313-71cd7b7295d5 \
	--conf spark.kubernetes.executor.podNamePrefix=kyuubi-kyuubi-user-spark-sql-apache-default-6724f97a-41b7-4a29-9313-71cd7b7295d5-6724f97a-41b7-4a29-9313-71cd7b7295d5 \
	--conf spark.master=k8s://https://10.100.0.1:443 \
	--conf spark.kubernetes.driverEnv.SPARK_USER_NAME=apache \
	--conf spark.executorEnv.SPARK_USER_NAME=apache \
	--proxy-user apache /opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.8.0.jar
23/11/13 09:26:13 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/11/13 09:26:13 INFO SignalRegister: Registering signal handler for TERM
23/11/13 09:26:13 INFO SignalRegister: Registering signal handler for HUP
23/11/13 09:26:13 INFO SignalRegister: Registering signal handler for INT
23/11/13 09:26:13 INFO HiveConf: Found configuration file null
23/11/13 09:26:14 INFO SparkContext: Running Spark version 3.4.1
23/11/13 09:26:14 INFO ResourceUtils: ==============================================================
23/11/13 09:26:14 INFO ResourceUtils: No custom resources configured for spark.driver.
23/11/13 09:26:14 INFO ResourceUtils: ==============================================================
23/11/13 09:26:14 INFO SparkContext: Submitted application: kyuubi_USER_SPARK_SQL_apache_default_6724f97a-41b7-4a29-9313-71cd7b7295d5
23/11/13 09:26:14 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
23/11/13 09:26:14 INFO ResourceProfile: Limiting resource is cpus at 1 tasks per executor
23/11/13 09:26:14 INFO ResourceProfileManager: Added ResourceProfile id: 0
23/11/13 09:26:14 INFO SecurityManager: Changing view acls to: kyuubi,apache
23/11/13 09:26:14 INFO SecurityManager: Changing modify acls to: kyuubi,apache
23/11/13 09:26:14 INFO SecurityManager: Changing view acls groups to:
23/11/13 09:26:14 INFO SecurityManager: Changing modify acls groups to:
23/11/13 09:26:14 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: kyuubi, apache; groups with view permissions: EMPTY; users with modify permissions: kyuubi, apache; groups with modify permissions: EMPTY
23/11/13 09:26:14 INFO Utils: Successfully started service 'sparkDriver' on port 42665.
23/11/13 09:26:14 INFO SparkEnv: Registering MapOutputTracker
23/11/13 09:26:14 INFO SparkEnv: Registering BlockManagerMaster
23/11/13 09:26:14 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
23/11/13 09:26:14 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
23/11/13 09:26:14 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
23/11/13 09:26:14 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-1f6ddf78-bc8d-47bb-9bba-a8d01df325d8
23/11/13 09:26:14 INFO MemoryStore: MemoryStore started with capacity 413.9 MiB
23/11/13 09:26:14 INFO SparkEnv: Registering OutputCommitCoordinator
23/11/13 09:26:14 INFO JettyUtils: Start Jetty 0.0.0.0:0 for SparkUI
23/11/13 09:26:14 INFO Utils: Successfully started service 'SparkUI' on port 41633.
23/11/13 09:26:14 INFO SparkContext: Added JAR file:/opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.8.0.jar at spark://172.31.28.171:42665/jars/kyuubi-spark-sql-engine_2.12-1.8.0.jar with timestamp 1699867574094
23/11/13 09:26:14 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
23/11/13 09:26:15 INFO ExecutorPodsAllocator: Going to request 2 executors from Kubernetes for ResourceProfile Id: 0, target: 2, known: 0, sharedSlotFromPendingPods: 2147483647.
23/11/13 09:26:15 WARN ExecutorPodsSnapshotsStoreImpl: Exception when notifying snapshot subscriber.
org.apache.spark.SparkException: Must specify the executor container image
	at org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStep.$anonfun$executorContainerImage$1(BasicExecutorFeatureStep.scala:44)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStep.<init>(BasicExecutorFeatureStep.scala:44)
	at org.apache.spark.scheduler.cluster.k8s.KubernetesExecutorBuilder.buildFromFeatures(KubernetesExecutorBuilder.scala:69)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$1(ExecutorPodsAllocator.scala:430)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.requestNewExecutors(ExecutorPodsAllocator.scala:417)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$36(ExecutorPodsAllocator.scala:370)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$36$adapted(ExecutorPodsAllocator.scala:363)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.onNewSnapshots(ExecutorPodsAllocator.scala:363)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$start$3(ExecutorPodsAllocator.scala:134)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$start$3$adapted(ExecutorPodsAllocator.scala:134)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsSnapshotsStoreImpl$SnapshotsSubscriber.org$apache$spark$scheduler$cluster$k8s$ExecutorPodsSnapshotsStoreImpl$SnapshotsSubscriber$$processSnapshotsInternal(ExecutorPodsSnapshotsStoreImpl.scala:143)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsSnapshotsStoreImpl$SnapshotsSubscriber.processSnapshots(ExecutorPodsSnapshotsStoreImpl.scala:131)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsSnapshotsStoreImpl.$anonfun$addSubscriber$1(ExecutorPodsSnapshotsStoreImpl.scala:85)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
23/11/13 09:26:15 INFO ExecutorPodsAllocator: Going to request 2 executors from Kubernetes for ResourceProfile Id: 0, target: 2, known: 0, sharedSlotFromPendingPods: 2147483647.
23/11/13 09:26:15 WARN ExecutorPodsSnapshotsStoreImpl: Exception when notifying snapshot subscriber.
org.apache.spark.SparkException: Must specify the executor container image
	at org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStep.$anonfun$executorContainerImage$1(BasicExecutorFeatureStep.scala:44)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStep.<init>(BasicExecutorFeatureStep.scala:44)
	at org.apache.spark.scheduler.cluster.k8s.KubernetesExecutorBuilder.buildFromFeatures(KubernetesExecutorBuilder.scala:69)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$1(ExecutorPodsAllocator.scala:430)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.requestNewExecutors(ExecutorPodsAllocator.scala:417)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$36(ExecutorPodsAllocator.scala:370)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$36$adapted(ExecutorPodsAllocator.scala:363)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.onNewSnapshots(ExecutorPodsAllocator.scala:363)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$start$3(ExecutorPodsAllocator.scala:134)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$start$3$adapted(ExecutorPodsAllocator.scala:134)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsSnapshotsStoreImpl$SnapshotsSubscriber.org$apache$spark$scheduler$cluster$k8s$ExecutorPodsSnapshotsStoreImpl$SnapshotsSubscriber$$processSnapshotsInternal(ExecutorPodsSnapshotsStoreImpl.scala:143)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsSnapshotsStoreImpl$SnapshotsSubscriber$$anon$2.run(ExecutorPodsSnapshotsStoreImpl.scala:163)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
23/11/13 09:26:15 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37109.
23/11/13 09:26:15 INFO NettyBlockTransferService: Server created on 172.31.28.171:37109
23/11/13 09:26:15 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
23/11/13 09:26:15 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 172.31.28.171, 37109, None)
23/11/13 09:26:15 INFO BlockManagerMasterEndpoint: Registering block manager 172.31.28.171:37109 with 413.9 MiB RAM, BlockManagerId(driver, 172.31.28.171, 37109, None)
23/11/13 09:26:15 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 172.31.28.171, 37109, None)
23/11/13 09:26:15 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 172.31.28.171, 37109, None)
23/11/13 09:26:16 INFO ExecutorPodsAllocator: Going to request 2 executors from Kubernetes for ResourceProfile Id: 0, target: 2, known: 0, sharedSlotFromPendingPods: 2147483647.
23/11/13 09:26:16 WARN ExecutorPodsSnapshotsStoreImpl: Exception when notifying snapshot subscriber.
org.apache.spark.SparkException: Must specify the executor container image
	at org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStep.$anonfun$executorContainerImage$1(BasicExecutorFeatureStep.scala:44)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStep.<init>(BasicExecutorFeatureStep.scala:44)
	at org.apache.spark.scheduler.cluster.k8s.KubernetesExecutorBuilder.buildFromFeatures(KubernetesExecutorBuilder.scala:69)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$1(ExecutorPodsAllocator.scala:430)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.requestNewExecutors(ExecutorPodsAllocator.scala:417)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$36(ExecutorPodsAllocator.scala:370)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$36$adapted(ExecutorPodsAllocator.scala:363)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.onNewSnapshots(ExecutorPodsAllocator.scala:363)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$start$3(ExecutorPodsAllocator.scala:134)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$start$3$adapted(ExecutorPodsAllocator.scala:134)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsSnapshotsStoreImpl$SnapshotsSubscriber.org$apache$spark$scheduler$cluster$k8s$ExecutorPodsSnapshotsStoreImpl$SnapshotsSubscriber$$processSnapshotsInternal(ExecutorPodsSnapshotsStoreImpl.scala:143)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsSnapshotsStoreImpl$SnapshotsSubscriber.processSnapshots(ExecutorPodsSnapshotsStoreImpl.scala:131)
	at org.apache.spark.scheduler.cluster.k8s.ExecutorPodsSnapshotsStoreImpl.$anonfun$addSubscriber$1(ExecutorPodsSnapshotsStoreImpl.scala:85)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

pan3793 · 2023-11-13T09:44:27Z

pan3793
Nov 13, 2023
Collaborator

Please follow the official Spark on K8s docs to set up Spark properly first. Specific to the errors you pasted, please add the following Spark configuration

spark.kubernetes.container.image=<spark-image>

9 replies

jseungtaek Nov 14, 2023
Author

I've read the spark on k8s doc many times. Please let me know if there's anything missing

pan3793 Nov 14, 2023
Collaborator

I'm not sure what the exact problem you encountered, the logs say "... in namespace [null]", which looks weird.

jseungtaek Nov 14, 2023
Author

Thank you for your interest.
I created a new Kubernetes cluster and ran kyuubi, and the following error occurred. Please reply.

kyuubi version = 1.7.3

Error: org.apache.kyuubi.KyuubiSQLException: Error operating LaunchEngine: java.io.IOException: Cannot run program "/opt/kyuubi/bin/spark-submit" (in directory "/opt/kyuubi/work/anonymous"): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.apache.kyuubi.engine.ProcBuilder.start(ProcBuilder.scala:206)
at org.apache.kyuubi.engine.ProcBuilder.start$(ProcBuilder.scala:205)
at org.apache.kyuubi.engine.spark.SparkProcessBuilder.start(SparkProcessBuilder.scala:38)
at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:202)
at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.tryWithLock(ZookeeperDiscoveryClient.scala:180)
at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:168)
at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:173)
at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:277)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:277)
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2(KyuubiSessionImpl.scala:147)
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2$adapted(KyuubiSessionImpl.scala:123)
at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36)
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:123)
at org.apache.kyuubi.session.KyuubiSession.handleSessionException(KyuubiSession.scala:49)
at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:123)
at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:60)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 22 more

at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
at org.apache.kyuubi.operation.KyuubiOperation$$anonfun$onError$1.$anonfun$applyOrElse$1(KyuubiOperation.scala:76)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.kyuubi.Utils$.withLockRequired(Utils.scala:395)
at org.apache.kyuubi.operation.AbstractOperation.withLockRequired(AbstractOperation.scala:51)
at org.apache.kyuubi.operation.KyuubiOperation$$anonfun$onError$1.applyOrElse(KyuubiOperation.scala:60)
at org.apache.kyuubi.operation.KyuubiOperation$$anonfun$onError$1.applyOrElse(KyuubiOperation.scala:57)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:62)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

Caused by: java.io.IOException: Cannot run program "/opt/kyuubi/bin/spark-submit" (in directory "/opt/kyuubi/work/anonymous"): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.apache.kyuubi.engine.ProcBuilder.start(ProcBuilder.scala:206)
at org.apache.kyuubi.engine.ProcBuilder.start$(ProcBuilder.scala:205)
at org.apache.kyuubi.engine.spark.SparkProcessBuilder.start(SparkProcessBuilder.scala:38)
at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:202)
at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.tryWithLock(ZookeeperDiscoveryClient.scala:180)
at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:168)
at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:173)
at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:277)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:277)
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2(KyuubiSessionImpl.scala:147)
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2$adapted(KyuubiSessionImpl.scala:123)
at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36)
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:123)
at org.apache.kyuubi.session.KyuubiSession.handleSessionException(KyuubiSession.scala:49)
at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:123)
at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:60)
... 5 more
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 22 more (state=,code=0)

pan3793 Nov 14, 2023
Collaborator

consider using apache/kyuubi:1.8.0-spark or apache/kyuubi:1.7.3-spark, those images contain the spark client

jseungtaek Nov 14, 2023
Author

the kyuubi pod image apply 1.7.3-spark
and
I'm trying to access via beeline --conf mark.Where can I put Kubernetes.container.image=apache/spark:3.3.3?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kyuubi initializing error #5681

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 9 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

kyuubi initializing error #5681

jseungtaek Nov 13, 2023

Replies: 1 comment · 9 replies

pan3793 Nov 13, 2023 Collaborator

jseungtaek Nov 14, 2023 Author

pan3793 Nov 14, 2023 Collaborator

jseungtaek Nov 14, 2023 Author

pan3793 Nov 14, 2023 Collaborator

jseungtaek Nov 14, 2023 Author

jseungtaek
Nov 13, 2023

Replies: 1 comment 9 replies

pan3793
Nov 13, 2023
Collaborator

jseungtaek Nov 14, 2023
Author

pan3793 Nov 14, 2023
Collaborator

jseungtaek Nov 14, 2023
Author

pan3793 Nov 14, 2023
Collaborator

jseungtaek Nov 14, 2023
Author