Building Docker image to run Spark on Ray with RAPIDS

[chenya-zhang]

Hey there!

We are trying to experiment Spark on Ray with RAPIDS but not sure if RAPIDS can support this case.

1. The example Dockerfile for spark-rapids k8s setup: https://nvidia.github.io/spark-rapids/docs/get-started/Dockerfile.cuda
2. The OSS project for Spark on Ray: https://github.com/oap-project/raydp

In 1), we find the below commands to copy items under spark/:

COPY spark/jars /opt/spark/jars
COPY spark/bin /opt/spark/bin
COPY spark/sbin /opt/spark/sbin
COPY spark/kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY spark/examples /opt/spark/examples
COPY spark/kubernetes/tests /opt/spark/tests
COPY spark/data /opt/spark/data

In 2), if running pip install raydp-nightly, we can find pyspark/.
Under pyspark/, it has the below content.

/anaconda3/lib/python3.7/site-packages/pyspark# ls
__init__.py      context.py                java_gateway.py  rdd.py             statcounter.py
__pycache__      daemon.py                 join.py          rddsampler.py      status.py
_globals.py      data                      licenses         resource           storagelevel.py
_typing.pyi      examples                  ml               resultiterable.py  streaming
accumulators.py  files.py                  mllib            sbin               taskcontext.py
bin              find_spark_home.py        pandas           serializers.py     traceback_utils.py
broadcast.py     install.py                profiler.py      shell.py           util.py
cloudpickle      instrumentation_utils.py  py.typed         shuffle.py         version.py
conf.py          jars                      python           sql                worker.py

In this case, will there be concerns if we instead COPY pyspark/jars /opt/pyspark/jars or set SPARK_HOME to the existing .../pyspark installed by RayDP, 2) there is no /kubernetes/dockerfiles/spark/entrypoint.sh or /kubernetes/tests under pyspark/- I think they may not be required if we are able to launch Spark with RayDP on k8s.

Any suggestions or pointers would be very helpful, thanks!

[jlowe]

We're not familiar with the Spark on Ray project and have never tested in that environment. As long as Spark on Ray still supports Spark plugins and Spark's GPU scheduling, it should work.

Most of the Dockerfile commands you're seeing are just to get Spark setup on Kubernetes in the container and isn't specific to the RAPIDS Accelerator. The key parts in the Dockerfile for the RAPIDS Accelerator rather than generic Spark setup are primarily starting from an image that provides CUDA and these commands:

COPY rapids-4-spark_2.12-*.jar /opt/sparkRapidsPlugin
COPY getGpusResources.sh /opt/sparkRapidsPlugin

They get the RAPIDS Accelerator jar and the GPU scheduling script in the container so they can be referenced by the spark submit commands referenced at https://github.com/NVIDIA/spark-rapids/blob/branch-23.02/docs/get-started/getting-started-kubernetes.md#running-spark-applications-in-the-kubernetes-cluster.

In summary, once you get Spark setup in a container that provides CUDA and can schedule for GPUs, you should only need to ensure the RAPIDS Accelerator jar is in the container and the relevant startup configs are specified with the Spark job to get the RAPIDS Accelerator to work (again, assuming Spark on Ray supports Spark's GPU scheduling and Spark plugins).

[chenya-zhang]

Thanks @jlowe for the helpful suggestions. Will let you know.

[chenya-zhang]

Updates:

1. We built a docker image on top of nvidia/cuda:11.5.0-devel-ubuntu18.04 and include ray, raydp, java, rapids-4-spark_2.12-23.02.0.jar and getGpusResources.sh.
2. After deploying the Ray cluster with this image, we have a Ray worker running with 8 nvidia gpu in the cluster.
3. If exec into the Ray worker and run nvidia-smi, we can see the below information:

 NVIDIA-SMI 470.161.03   Driver Version: 470.161.03   CUDA Version: N/A      |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A100-SXM...  On   | 00000000:10:1C.0 Off |                    0 |
| N/A   22C    P0    50W / 400W |      0MiB / 40536MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA A100-SXM...  On   | 00000000:10:1D.0 Off |                    0 |
| N/A   21C    P0    47W / 400W |      0MiB / 40536MiB |      0%      Default |
|                               |                      |             Disabled 

4. Please note that the CUDA version is N/A, not sure if it can be an issue.
5. We then use RayDP to launch a simple Spark session. We have tested RayDP is able to support the configs option before.

spark = raydp.init_spark(app_name='raydp_pyspark_example',
                         num_executors=1,
                         executor_cores=1,
                         executor_memory='1GB',
                         configs={
                             "spark.executor.resource.gpu.amount": "1",
                             "spark.executor.resource.gpu.vendor": "nvidia.com",
                             "spark.executor.resource.gpu.discoveryScript": "/home/ray/anaconda3/lib/python3.7/site-packages/pyspark/jars/getGpusResources.sh"
                         })

Unfortunately, there we got the error that No available node types can fulfill resource request {'gpu': 1.0, 'CPU': 1.0, 'memory': 1073741824.0}. Add suitable node types to this cluster to resolve this issue.

If there is anything else we can do to verify the gpu resource, that would be super helpful.
Thanks!

[chenya-zhang]

oh I misunderstood you!
I still used the CUDA base image but add ray, raydp from my cpu image to it.

To install the correct CUDA runtime, shall I follow the below link for 11.5.0-devel-ubuntu18.04?

https://developer.nvidia.com/cuda-11-5-0-download-archive?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=18.04&target_type=deb_local

[jlowe]

Yes, you could follow the installation instructions for Debian packages either local or network in your Dockerfile.

[jlowe]

Any update on trying to add the CUDA toolkit to the functioning CPU image for Ray? Curious if you were able to make this work.

[avinashahuja]

@chenya-zhang: Is this resolved? Can we mark this as closed? Would you be willing to share the steps that helped you resolve this?

[chenya-zhang]

@jlowe @avinashahuja hey folks, we are focusing on Spark Rapids GPU experiments at this moment. We can resume the discussion on this thread later.

[jlowe]

We were able to reproduce the No available node types can fulfill resource request problem, and I think we found a solution. The ray status command gave a clue:

Resources
---------------------------------------------------------------
Usage:
 0.0/32.0 CPU
 0.0/2.0 GPU
 0B/128.00GiB memory
 0B/35.78GiB object_store_memory

Demands:
 {'CPU': 1.0, 'gpu': 1.0, 'memory': 1073741824.0}: 1+ pending tasks/actors

Note that the cluster is reporting CPU, GPU, and memory resources but receiving a request for CPU, gpu, and memory resources. Note the case difference of GPU vs. gpu. We were able to get the cluster to successfully allocate a GPU container with the following updates:

• changed the gpuGpusResources.sh script to use \"GPU\" instead of \"gpu\"
• changed the spark.executor.resource.gpu.* configs to corresponding spark.executor.resource.GPU.* configs
• changed spark.task.resource.gpu.amount to spark.task.resource.GPU.amount

These changes caused Spark to start asking for GPU resources instead of gpu resources.