spark-ShuffleWriter.adoc

ShuffleWriter

Caution

FIXME

ShuffleWriter Contract

abstract class ShuffleWriter[K, V] {
  def write(records: Iterator[Product2[K, V]]): Unit
  def stop(success: Boolean): Option[MapStatus]
}

Note	ShuffleWriter is a private[spark] contract.

Table 1. ShuffleWriter Contract

Method	Description
write	Writes a sequence of records (for a RDD partition) to a shuffle system when a ShuffleMapTask writes its execution result.
stop	Closes a ShuffleWriter and returns MapStatus if the writing completed successfully. Used when a ShuffleMapTask finishes execution with the input success flag to match the status of the task execution.