spark-BlockManager-BlockManagerSource.adoc

BlockManagerSource — Metrics Source for BlockManager

BlockManagerSource is the metrics source of a BlockManager.

BlockManagerSource is registered under the name BlockManager (when SparkContext is created).

Table 1. BlockManagerSource’s Gauge Metrics (in alphabetical order)

Name	Type	Description
disk.diskSpaceUsed_MB	long	Requests BlockManagerMaster for storage status (for every BlockManager) and sums up their disk space used (diskUsed).
memory.maxMem_MB	long	Requests BlockManagerMaster for storage status (for every BlockManager) and sums up their maximum memory limit (maxMem).
memory.maxOffHeapMem_MB	long	Requests BlockManagerMaster for storage status (for every BlockManager) and sums up their off-heap memory remaining (maxOffHeapMem).
memory.maxOnHeapMem_MB	long	Requests BlockManagerMaster for storage status (for every BlockManager) and sums up their on-heap memory remaining (maxOnHeapMem).
memory.memUsed_MB	long	Requests BlockManagerMaster for storage status (for every BlockManager) and sums up their memory used (memUsed).
memory.offHeapMemUsed_MB	long	Requests BlockManagerMaster for storage status (for every BlockManager) and sums up their off-heap memory used (offHeapMemUsed).
memory.onHeapMemUsed_MB	long	Requests BlockManagerMaster for storage status (for every BlockManager) and sums up their on-heap memory used (onHeapMemUsed).
memory.remainingMem_MB	long	Requests BlockManagerMaster for storage status (for every BlockManager) and sums up their memory remaining (memRemaining).
memory.remainingOffHeapMem_MB	long	Requests BlockManagerMaster for storage status (for every BlockManager) and sums up their off-heap memory remaining (offHeapMemRemaining).
memory.remainingOnHeapMem_MB	long	Requests BlockManagerMaster for storage status (for every BlockManager) and sums up their on-heap memory remaining (onHeapMemRemaining).

You can access the BlockManagerSource metrics using the web UI’s port (as spark.ui.port configuration property).

$ http --follow http://localhost:4040/metrics/json \
    | jq '.gauges | keys | .[] | select(test(".driver.BlockManager"; "g"))'
"local-1528725411625.driver.BlockManager.disk.diskSpaceUsed_MB"
"local-1528725411625.driver.BlockManager.memory.maxMem_MB"
"local-1528725411625.driver.BlockManager.memory.maxOffHeapMem_MB"
"local-1528725411625.driver.BlockManager.memory.maxOnHeapMem_MB"
"local-1528725411625.driver.BlockManager.memory.memUsed_MB"
"local-1528725411625.driver.BlockManager.memory.offHeapMemUsed_MB"
"local-1528725411625.driver.BlockManager.memory.onHeapMemUsed_MB"
"local-1528725411625.driver.BlockManager.memory.remainingMem_MB"
"local-1528725411625.driver.BlockManager.memory.remainingOffHeapMem_MB"
"local-1528725411625.driver.BlockManager.memory.remainingOnHeapMem_MB"

BlockManagerSource takes a BlockManager when created.

BlockManagerSource is created when SparkContext is created.