This document assumes you have YuniKorn and its admission-controller both installed. Please refer to get started to see how that is done.
Prepare the docker image for Spark
To run Spark on Kubernetes, you'll need the Spark docker images. You can 1) use the docker images provided by the YuniKorn team, or 2) build one from scratch. If you want to build your own Spark docker image, you can
- Download a Spark version that has Kubernetes support, URL: https://github.com/apache/spark
- Build spark with Kubernetes support:
Create a namespace for Spark jobs
Create a namespace:
Create service account and cluster role bindings under
Do NOT use
ClusterRoleBinding to run Spark jobs in production, please configure a more fine-grained
security context for running Spark jobs. See more about how to configure proper RBAC rules here.
Submit a Spark job
If this is running from local machine, you will need to start the proxy in order to talk to the api-server.
Run a simple SparkPi job (this assumes that the Spark binaries are installed to
You'll see Spark driver and executors been created on Kubernetes:
You can also view the job info from YuniKorn UI. If you do not know how to access the YuniKorn UI, please read the document here.
What happens behind the scenes?
When the Spark job is submitted to the cluster, the job is submitted to
spark-test namespace. The Spark driver pod will
be firstly created under this namespace. Since this cluster has YuniKorn admission-controller enabled, when the driver pod
get created, the admission-controller mutates the pod's spec and injects
schedulerName=yunikorn, by doing this, the
default K8s scheduler will skip this pod and it will be scheduled by YuniKorn instead. See how this is done by configuring
another scheduler in Kubernetes.
The default configuration has placement rule enabled, which automatically maps the
spark-test namespace to a YuniKorn
root.spark-test. All Spark jobs submitted to this namespace will be automatically submitted to the queue first.
To see more about how placement rule works, please see doc placement-rules. By far,
the namespace defines the security context of the pods, and the queue determines how the job and pods will be scheduled
with considering of job ordering, queue resource fairness, etc. Note, this is the simplest setup, which doesn't enforce
the queue capacities. The queue is considered as having unlimited capacity.
YuniKorn reuses the Spark application ID set in label
spark-app-selector, and this job is submitted
to YuniKorn and being considered as a job. The job is scheduled and running as there is sufficient resources in the cluster.
YuniKorn allocates the driver pod to a node, binds the pod and starts all the containers. Once the driver pod gets started,
it requests for a bunch of executor pods to run its tasks. Those pods will be created in the same namespace as well and
scheduled by YuniKorn as well.