spark-core-AppStatusListener.adoc

AppStatusListener

AppStatusListener is a SparkListener that AppStatusStore uses to…​FIXME

AppStatusListener is created when:

• AppStatusStore is requested to createLiveStore (with the live flag enabled)

• FsHistoryProvider is requested to rebuildAppStore (with the live flag disabled)

Table 1. AppStatusListener’s Event Handlers

Event	Handler
SparkListenerApplicationStart	onApplicationStart
SparkListenerApplicationEnd	onApplicationEnd
SparkListenerBlockManagerAdded	onBlockManagerAdded
SparkListenerBlockManagerRemoved	onBlockManagerRemoved
SparkListenerBlockUpdated	onBlockUpdated
SparkListenerEnvironmentUpdate	onEnvironmentUpdate
SparkListenerEvent	onOtherEvent
SparkListenerExecutorAdded	onExecutorAdded
SparkListenerExecutorBlacklisted	onExecutorBlacklisted
SparkListenerExecutorMetricsUpdate	onExecutorMetricsUpdate
SparkListenerExecutorRemoved	onExecutorRemoved
SparkListenerExecutorUnblacklisted	onExecutorUnblacklisted
SparkListenerJobStart	onJobStart
SparkListenerJobEnd	onJobEnd
SparkListenerNodeBlacklisted	onNodeBlacklisted
SparkListenerNodeUnblacklisted	onNodeUnblacklisted
SparkListenerStageCompleted	onStageCompleted
SparkListenerStageSubmitted	onStageSubmitted
SparkListenerTaskEnd	onTaskEnd
SparkListenerTaskGettingResult	onTaskGettingResult
SparkListenerTaskStart	onTaskStart
SparkListenerUnpersistRDD	onUnpersistRDD

Table 2. AppStatusListener’s Internal Properties (e.g. Registries, Counters and Flags)

Name	Description
appInfo	v1.ApplicationInfo
appSummary	AppSummary
liveUpdatePeriodNs
coresPerTask	Default: 1
liveRDDs	LiveRDDs by RDD ID
liveStages	LiveStages by (Int, Int)
liveTasks	LiveTask by task ID
liveJobs	LiveJob by job ID
liveExecutors	LiveExecutor by executor ID
pools	SchedulerPool by FIXME
activeExecutorCount	Number of active executors

onStageSubmitted Method

onStageSubmitted(event: SparkListenerStageSubmitted): Unit

Note	onStageSubmitted is part of SparkListener Contract to…​FIXME.

onStageSubmitted…​FIXME

update Internal Method

update(entity: LiveEntity, now: Long, last: Boolean = false): Unit

update simply requests the LiveEntity to write (with the ElementTrackingStore as the store and the last flag as checkTriggers flag).

Note	update is used in event handlers (i.e. onApplicationStart, onExecutorRemoved, onJobEnd, onStageSubmitted, onTaskEnd, onStageCompleted), liveUpdate, maybeUpdate, flush and updateRDDBlock.

flush Internal Method

flush(): Unit

flush…​FIXME

Note	flush is used when…​FIXME

maybeUpdate Internal Method

maybeUpdate(entity: LiveEntity, now: Long): Unit

maybeUpdate…​FIXME

Note	maybeUpdate is used when…​FIXME

liveUpdate Internal Method

liveUpdate(entity: LiveEntity, now: Long): Unit

liveUpdate…​FIXME

Note	liveUpdate is used when…​FIXME

updateStreamBlock Internal Method

updateStreamBlock(event: SparkListenerBlockUpdated, stream: StreamBlockId): Unit

updateStreamBlock…​FIXME

Note	updateStreamBlock is used exclusively when AppStatusListener is requested to handle a SparkListenerBlockUpdated event (for a StreamBlockId).

Intercepting SparkListenerBlockUpdated Events — onBlockUpdated Handler Method

onBlockUpdated(event: SparkListenerBlockUpdated): Unit

Note	onBlockUpdated is part of SparkListener Contract to…​FIXME.

onBlockUpdated simply dispatches to the following event-specific handlers (per BlockId type):

• updateRDDBlock for RDDBlockIds

• updateStreamBlock for StreamBlockIds

• Ignores (swallows) the SparkListenerBlockUpdated event for the other types

Creating AppStatusListener Instance

AppStatusListener takes the following when created:

• ElementTrackingStore

• SparkConf

• live flag

• Optional lastUpdateTime (default: None)

AppStatusListener initializes the internal registries and counters.

updateRDDBlock Internal Method

updateRDDBlock(event: SparkListenerBlockUpdated, block: RDDBlockId): Unit

updateRDDBlock…​FIXME

Note	updateRDDBlock is used exclusively when AppStatusListener is requested to handle a SparkListenerBlockUpdated event (for a RDDBlockId).