Skip to content

Latest commit

 

History

History
24 lines (21 loc) · 761 Bytes

README.md

File metadata and controls

24 lines (21 loc) · 761 Bytes
Raw

Spark Batch Base

Overview

This is a skeleton/example application for a spark batch application

Building

  • To build and run the tests sbt clean test
  • To build the deployable application package sbt clean assembly

Running

You can run the project on an existing Spark cluster. An easy way to start one is provided in deploy/spark-cluster.yml

  1. Start the Spark cluster.
deploy/start-spark.sh
  1. Verify that Spark Master and Spark worker are running and connected via the Spark Admin Console
  2. Launch the application
$SPARK_HOME/bin/spark-submit \
    --class Main \
    --master spark://localhost:7077 \
    ./target/scala-2.11/spark-batch-base-assembly-0.0.1-SNAPSHOT.jar \
    --env default