[Bug] The USER share level engine seems to have a memory leak

[wForget]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

The USER share level engine seems to have a memory leak.

Affects Version(s)

1.4.0

Kyuubi Server Log Output

No response

Kyuubi Engine Log Output

22/05/30 11:12:09 INFO ExecuteStatement: 
           Spark application name: kyuubi_USER_SPARK_SQL_test_default_21d23da7-8fb5-4f05-afb8-fcfccc32017b
                 application ID: application_1652688933831_889397
                 application web UI: http://XXX:8808/proxy/application_1652688933831_889397
                 master: yarn
                 deploy mode: cluster
                 version: 3.1.1
           Start time: 2022-05-29T04:50:31.352
           User: test
           Submitted application application_1652688933831_889397
22/05/30 11:12:20 INFO AsyncEventQueue: Process of event SparkStatementEvent(cloud_service,0096e875-20be-4ee7-83d3-76d56528a6be,SELECT ENGINE_ID(),application_1652688933831_889397,c8d25743-5fc1-460c-aea1-a8f8ea8ffa38,1653880329742,RUNNING,1653880329784,-1,,) by listener EventLoggingListener took 11.018725009s.
22/05/30 11:12:20 INFO SQLOperationListener: Query [bc4df081-6ddf-40cb-85dc-f8bb68224a8d]: Stage 4058 started with 1 tasks, 1 active stages running
22/05/30 11:12:20 INFO YarnClusterScheduler: Cancelling stage 4058
22/05/30 11:12:20 INFO YarnClusterScheduler: Killing all running tasks in stage 4058: Stage cancelled
22/05/30 11:12:20 INFO SparkThriftBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
22/05/30 11:12:20 INFO SparkSQLSessionManager: Opening session for ***@***
22/05/30 11:12:20 WARN SparkSQLSessionManager: Cannot modify the value of a Spark config: spark.yarn.queue
22/05/30 11:12:20 WARN SparkSQLSessionManager: Cannot modify the value of a Spark config: spark.scheduler.mode
22/05/30 11:12:20 WARN SparkSQLSessionManager: Cannot modify the value of a Spark config: spark.executor.instances
22/05/30 11:12:20 INFO DAGScheduler: ResultStage 4058 (collect at ExecuteStatement.scala:111) failed in 23.537 s due to Job aborted due to stage failure: Task serialization failed: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
	at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
	at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
	at org.apache.spark.broadcast.TorrentBroadcast$.$anonfun$blockifyObject$1(TorrentBroadcast.scala:314)
	at org.apache.spark.broadcast.TorrentBroadcast$.$anonfun$blockifyObject$1$adapted(TorrentBroadcast.scala:314)
	at org.apache.spark.broadcast.TorrentBroadcast$$$Lambda$2692/41914094.apply(Unknown Source)
	at org.apache.spark.util.io.ChunkedByteBufferOutputStream.allocateNewChunkIfNeeded(ChunkedByteBufferOutputStream.scala:87)
	at org.apache.spark.util.io.ChunkedByteBufferOutputStream.write(ChunkedByteBufferOutputStream.scala:75)
	at net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:223)
	at net.jpountz.lz4.LZ4BlockOutputStream.finish(LZ4BlockOutputStream.java:255)
	at net.jpountz.lz4.LZ4BlockOutputStream.close(LZ4BlockOutputStream.java:193)
	at com.esotericsoftware.kryo.io.Output.close(Output.java:199)
	at org.apache.spark.serializer.KryoSerializationStream.close(KryoSerializer.scala:273)
	at org.apache.spark.broadcast.TorrentBroadcast$.$anonfun$blockifyObject$5(TorrentBroadcast.scala:321)
	at org.apache.spark.broadcast.TorrentBroadcast$$$Lambda$2725/723036004.apply$mcV$sp(Unknown Source)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1448)
	at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:321)
	at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:138)
	at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:91)
	at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:35)
	at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:77)
	at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1509)
	at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1388)
	at org.apache.spark.scheduler.DAGScheduler.submitStage(DAGScheduler.scala:1226)
	at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:1168)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2395)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2387)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2376)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)

Kyuubi Server Configurations

No response

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[wForget]

referenced by the org.apache.hadoop.conf.Configuration object in org.apache.iceberg.spark.SparkCatalog

[wForget]

#2250 support to limit the spark engine max running time, so that we can periodically restart the shared engine to alleviate this issue.

[wForget]

Caused by SPARK-34087, fixed in apache/spark#31839