Skip to content
This repository has been archived by the owner on Jan 9, 2020. It is now read-only.

In-cluster client mode #456

Open
wants to merge 5 commits into
base: branch-2.2-kubernetes
Choose a base branch
from
Open

In-cluster client mode #456

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
872b4c2
Enabling in-cluster client mode execution
Aug 23, 2017
2fba486
Usage docs
Aug 23, 2017
acca24a
Rewording in-cluster client mode docs
Aug 24, 2017
9e831a1
Merge branch 'branch-2.2-kubernetes' of https://github.com/apache-spa…
Aug 29, 2017
08b5433
Incorporating feedback into in-cluster docs
Aug 29, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -344,8 +344,8 @@ object SparkSubmit extends CommandLineUtils {

// The following modes are not supported or applicable
(clusterManager, deployMode) match {
case (KUBERNETES, CLIENT) =>
printErrorAndExit("Client mode is currently not supported for Kubernetes.")
case (KUBERNETES, CLIENT) if !inK8sCluster() =>
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree it's probably best to limit this to in-cluster now, but generally speaking I don't see why we shouldn't allow client mode with the appropriate networking caveats. Some network configurations allow the pod network to be fully routable (e.g. Calico).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure how difficult it would be to detect pod network connectivity upon application submission. The logic would probably have to abstracted out to the cluster manager or one of the initial steps, and in that case, we would have to throw a validation error much later on in the process as we would need to allow client mode through unobstructed in SparkSubmit. Definitely feasible though.

printErrorAndExit("Kubernetes currently only supports in-cluster client mode.")
case (KUBERNETES, CLUSTER) if args.isR =>
printErrorAndExit("Kubernetes does not currently support R applications.")
case (STANDALONE, CLUSTER) if args.isPython =>
Expand Down Expand Up @@ -856,6 +856,14 @@ object SparkSubmit extends CommandLineUtils {
res == SparkLauncher.NO_RESOURCE
}

/**
* Return whether the submission environment is within a Kubernetes cluster
*/
private[deploy] def inK8sCluster(): Boolean = {
!sys.env.get("KUBERNETES_SERVICE_HOST").isEmpty &&
!sys.env.get("KUBERNETES_SERVICE_PORT").isEmpty
}

/**
* Merge a sequence of comma-separated file lists, some of which may be null to indicate
* no files, into a single comma-separated string.
Expand Down
49 changes: 40 additions & 9 deletions docs/running-on-kubernetes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ For example, if the registry host is `registry-host` and the registry is listeni
docker push registry-host:5000/spark-driver:latest
docker push registry-host:5000/spark-executor:latest
docker push registry-host:5000/spark-init:latest

Note that `spark-base` is the base image for the other images. It must be built first before the other images, and then afterwards the other images can be built in any order.

## Submitting Applications to Kubernetes
Expand Down Expand Up @@ -196,10 +196,10 @@ is currently supported.

### Running PySpark

Running PySpark on Kubernetes leverages the same spark-submit logic when launching on Yarn and Mesos.
Python files can be distributed by including, in the conf, `--py-files`
Running PySpark on Kubernetes leverages the same spark-submit logic when launching on Yarn and Mesos.
Python files can be distributed by including, in the conf, `--py-files`

Below is an example submission:
Below is an example submission:


```
Expand Down Expand Up @@ -254,6 +254,37 @@ the command may then look like the following:

## Advanced

### Running in-cluster client mode applications

While Spark on Kubernetes does not support client mode applications, such as the PySpark shell, when launched from outside Kubernetes, Spark on Kubernetes does support client mode applications launched from within the cluster. This _in-cluster_ client mode bypasses some of the networking and dependency issues inherent to running a client from outside of a cluster while allowing much of the same functionality in terms of interactive use cases, such as the PySpark shell and Jupyter notebooks.

In order to run in client mode, use `kubectl attach` to attach to an existing driver pod on the cluster, or the following to run a new driver:

kubectl run -it --image=<driver image> --restart=Never -- /bin/bash
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use kubespark/spark-driver:latest as image in the commandline?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did not want to specify, but I see the benefit.


This will open up a shell into the specified driver pod from which you can run client mode applications. In order to appropriately configure
these in-cluster applications, be sure to set the following configuration value for all applications, as in the following `spark-submit` example,
which tells the cluster manager to refer back to the current driver pod as the driver for any applications you submit:

spark.kubernetes.driver.pod.name=$HOSTNAME

With that set, you should be able to run the following example from within the pod:

bin/spark-submit \
--deploy-mode client \
--class org.apache.spark.examples.SparkPi \
--master k8s://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT \
--kubernetes-namespace default \
--conf spark.app.name=spark-pi \
--conf spark.kubernetes.driver.pod.name=$HOSTNAME \
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if pods can infer their own name via the environment?

--conf spark.kubernetes.driver.docker.image=kubespark/spark-driver:latest \
--conf spark.kubernetes.executor.docker.image=kubespark/spark-executor:latest \
--conf spark.dynamicAllocation.enabled=true \
--conf spark.shuffle.service.enabled=true \
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel perhaps we should leave
`--conf spark.dynamicAllocation.enabled=true \

  •  --conf spark.shuffle.service.enabled=true

` out of the example? these are not default or required when running k8s right?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed.

--conf spark.kubernetes.shuffle.namespace=default \
--conf spark.kubernetes.shuffle.labels="app=spark-shuffle-service,spark-version=2.1.0" \
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2.2.0?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, will change later today.

local:///opt/spark/examples/jars/spark_examples_2.11-2.2.0.jar 10

### Securing the Resource Staging Server with TLS

The default configuration of the resource staging server is not secured with TLS. It is highly recommended to configure
Expand Down Expand Up @@ -773,25 +804,25 @@ from the other deployment modes. See the [configuration page](configuration.html
</td>
</tr>
<tr>
<td><code>spark.kubernetes.node.selector.[labelKey]</code></td>
<td><code>spark.kubernetes.node.selector.[labelKey]</code></td>
<td>(none)</td>
<td>
Adds to the node selector of the driver pod and executor pods, with key <code>labelKey</code> and the value as the
Adds to the node selector of the driver pod and executor pods, with key <code>labelKey</code> and the value as the
configuration's value. For example, setting <code>spark.kubernetes.node.selector.identifier</code> to <code>myIdentifier</code>
will result in the driver pod and executors having a node selector with key <code>identifier</code> and value
will result in the driver pod and executors having a node selector with key <code>identifier</code> and value
<code>myIdentifier</code>. Multiple node selector keys can be added by setting multiple configurations with this prefix.
</td>
</tr>
<tr>
<td><code>spark.executorEnv.[EnvironmentVariableName]</code></td>
<td><code>spark.executorEnv.[EnvironmentVariableName]</code></td>
<td>(none)</td>
<td>
Add the environment variable specified by <code>EnvironmentVariableName</code> to
the Executor process. The user can specify multiple of these to set multiple environment variables.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.driverEnv.[EnvironmentVariableName]</code></td>
<td><code>spark.kubernetes.driverEnv.[EnvironmentVariableName]</code></td>
<td>(none)</td>
<td>
Add the environment variable specified by <code>EnvironmentVariableName</code> to
Expand Down