Run RayCluster

Note

This example demonstrates how to set up KubeRay and run a RayCluster with the YuniKorn scheduler. It relies on an admission controller to configure the default applicationId and queue name. If you want more details, please refer to Yunikorn supported labels and Yunikorn queue setting.

Modify YuniKorn settings

Follow YuniKorn install guide and modify YuniKorn configmap "yunikorn-defaults" to allow ray operator based on k8s service account.

kubectl patch configmap yunikorn-defaults -n yunikorn --patch '{"data":{"admissionController.accessControl.systemUsers": "^system:serviceaccount:kube-system:,^system:serviceaccount:default:"}}' 

Setup a KubeRay operator

helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm repo update
helm install kuberay-operator kuberay/kuberay-operator --version 1.1.1

• The result should be as shown below

Create RayCluster

helm install raycluster kuberay/ray-cluster --version 1.1.1

• RayCluster result
• YuniKorn UI

Configure your Ray CRD(optional)

If you disable admission controller, you need to add the schedulerName: yunikorn in raycluster labels.

#example
metadata:
  labels:
    applicaionId: ray-<service_type>-0001
    queue: root.ray.<specific_queue_name>
spec:
  schedulerName: yunikorn # k8s will inform yunikorn based on this

Submit a RayJob to RayCluster

export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)
echo $HEAD_POD

kubectl exec -it $HEAD_POD -- python -c "import ray; ray.init(); print(ray.cluster_resources())"

Services in Kubernetes aren't directly accessible by default. However, you can use port-forwarding to connect to them locally.

kubectl port-forward service/raycluster-kuberay-head-svc 8265:8265

After port-forward set up, you can access the Ray dashboard by going to http://localhost:8265 in your web browser.

• Ray Dashboard