spark-deployment-environments.adoc

Deployment Environments — Run Modes

Spark Deployment Environments (aka Run Modes):

• local

• clustered

  • Spark Standalone

  • Spark on Apache Mesos

  • Spark on Hadoop YARN

A Spark application is composed of the driver and executors that can run locally (on a single JVM) or using cluster resources (like CPU, RAM and disk that are managed by a cluster manager).

Note	You can specify where to run the driver using the deploy mode (using --deploy-mode option of spark-submit or spark.submit.deployMode Spark property).

Master URLs

Spark supports the following master URLs (see private object SparkMasterRegex):

• local, local[N] and local[*] for Spark local

• local[N, maxRetries] for Spark local-with-retries

• local-cluster[N, cores, memory] for simulating a Spark cluster of N executors (threads), cores CPUs and memory locally (aka Spark local-cluster)

• spark://host:port,host1:port1,…​ for connecting to Spark Standalone cluster(s)

• mesos:// for Spark on Mesos cluster

• yarn for Spark on YARN

You can specify the master URL of a Spark application as follows:

1. spark-submit’s --master command-line option,

2. spark.master Spark property,

3. When creating a SparkContext (using setMaster method),

4. When creating a SparkSession (using master method of the builder interface).