为什么CONNECTION级别的spark engine每次退出yarn状态都是kill #4220

pushiwuhua7 · 2023-02-01T02:46:12Z

pushiwuhua7
Feb 1, 2023

kyuubi 1.6.1
spark 3.2.3

#spark.master=yarn;spark.submit.deployMode=cluster;kyuubi.engine.share.level=CONNECTION'

spark driver log

23/02/01 10:38:44 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [c181e4f5-bb0a-4012-92da-1a464ab265a4]
23/02/01 10:38:44 INFO SparkSQLSessionManager: SessionHandle [c181e4f5-bb0a-4012-92da-1a464ab265a4] is closed, current opening sessions 0
23/02/01 10:38:44 INFO SparkSQLSessionManager: Session stopped due to shared level is Connection.
23/02/01 10:38:44 INFO SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping.
23/02/01 10:38:44 INFO SparkTBinaryFrontendService: Service: [EngineServiceDiscovery] is stopping.
23/02/01 10:38:44 WARN ZookeeperDiscoveryClient: This Kyuubi instance *:46360 is now de-registered from ZooKeeper. The server will be shut down after the last client session completes.
23/02/01 10:38:44 INFO EngineServiceDiscovery: Clean up discovery service due to this is connection share level.
23/02/01 10:38:44 INFO CuratorFrameworkImpl: backgroundOperationsLoop exiting
23/02/01 10:38:44 INFO ZooKeeper: Session: 0x28598529bddd51b closed
23/02/01 10:38:44 INFO ClientCnxn: EventThread shut down for session: 0x28598529bddd51b
23/02/01 10:38:44 INFO EngineServiceDiscovery: Service[EngineServiceDiscovery] is stopped.
23/02/01 10:38:44 INFO SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is stopped.
23/02/01 10:38:44 INFO SparkTBinaryFrontendService: SparkTBinaryFrontend has stopped
23/02/01 10:38:44 INFO SparkSQLEngine: Service: [SparkSQLBackendService] is stopping.
23/02/01 10:38:44 INFO SparkSQLBackendService: Service: [SparkSQLSessionManager] is stopping.
23/02/01 10:38:44 INFO SparkSQLSessionManager: Service: [SparkSQLOperationManager] is stopping.
23/02/01 10:38:44 INFO SparkSQLOperationManager: Service[SparkSQLOperationManager] is stopped.
23/02/01 10:38:44 INFO SparkSQLSessionManager: Service[SparkSQLSessionManager] is stopped.
23/02/01 10:38:44 INFO SparkSQLBackendService: Service[SparkSQLBackendService] is stopped.
23/02/01 10:38:44 INFO SparkSQLEngine: Service[SparkSQLEngine] is stopped.
23/02/01 10:38:44 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [c181e4f5-bb0a-4012-92da-1a464ab265a4]
23/02/01 10:38:44 INFO SparkUI: Stopped Spark web UI at http://*:36584
23/02/01 10:38:44 INFO YarnClusterSchedulerBackend: Shutting down all executors
23/02/01 10:38:44 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
23/02/01 10:38:44 ERROR SparkSQLEngine: RECEIVED SIGNAL 15: TERM
23/02/01 10:38:44 ERROR ApplicationMaster: RECEIVED SIGNAL TERM
23/02/01 10:38:44 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

kyuubi log

2023-02-01 10:38:42.369 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[98466918-e2c9-4c0f-8026-f7f8779ada5a] in FINISHED_STATE
2023-02-01 10:38:42.369 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing xdc's query[98466918-e2c9-4c0f-8026-f7f8779ada5a]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.284 seconds
2023-02-01 10:38:42.370 INFO org.apache.kyuubi.credentials.HadoopCredentialsManager: Send new credentials with epoch 0 to SQL engine through session c7cad556-252a-4d4f-9a87-a8b6e3071f9c
2023-02-01 10:38:42.393 INFO org.apache.kyuubi.credentials.HadoopCredentialsManager: Update session credentials epoch from -1 to 0
2023-02-01 10:38:42.547 INFO org.apache.kyuubi.client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:6C 4A BB FB 52 3E 43 99 B5 AF 3C E6 26 43 06 5E, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engine side
2023-02-01 10:38:44.097 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Received request of closing SessionHandle [c7cad556-252a-4d4f-9a87-a8b6e3071f9c]
2023-02-01 10:38:44.098 INFO org.apache.kyuubi.session.KyuubiSessionManager: SessionHandle [c7cad556-252a-4d4f-9a87-a8b6e3071f9c] is closed, current opening sessions 0
2023-02-01 10:38:44.189 INFO org.apache.kyuubi.engine.ProcBuilder: Destroy the process, since waitCompletion is false.
2023-02-01 10:38:44.425 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl: Killed application application_1647872176667_5651252
2023-02-01 10:38:44.426 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Finished closing SessionHandle [c7cad556-252a-4d4f-9a87-a8b6e3071f9c]