Dev Environment Setup
There are several ways to setup a local development environment for Kubernetes, the three most common ones are
Minikube provisions a local Kubernetes cluster on several Virtual Machines (via VirtualBox or something similar).
docker-desktop on the other hand, sets up Kubernetes cluster in docker containers.
kind provides lightweight Kubernetes clusters for Windows, Linux and Mac.
Local Kubernetes cluster using Docker Desktop
In this tutorial, we will base all the installs on Docker Desktop. Even in this case we can use a lightweight minikube setup which gives the same functionality with less impact.
Download and install Docker-Desktop on your laptop. Latest version has an embedded version of Kubernetes so no additional install is needed. Just simply follow the instruction here to get Kubernetes up and running within docker-desktop.
Once Kubernetes is started in docker desktop, you should see something similar below:
This means that:
- Kubernetes is running.
- the command line tool
kubctlis installed in the
- the Kubernetes context is set to
Deploy and access dashboard
After setting up the local Kubernetes you need to deploy the dashboard using the following steps:
- follow the instructions in Kubernetes dashboard doc to deploy the dashboard.
- start the Kubernetes proxy in the background from a terminal to get access on the dashboard on the local host:
kubectl proxy &
- access the dashboard at the following URL: clickable link
Access local Kubernetes cluster
The dashboard as deployed in the previous step requires a token or config to sign in. Here we use the token to sign in. The token is generated automatically and can be retrieved from the system.
- retrieve the name of the dashboard token:
kubectl -n kube-system get secret | grep kubernetes-dashboard-token
- retrieve the content of the token, note that the token name ends with a random 5 character code and needs to be replaced with the result of step 1. As an example:
kubectl -n kube-system describe secret kubernetes-dashboard-token-tf6n8
- copy the token value which is part of the
Datasection with the tag
- select the Token option in the dashboard web UI:
- paste the token value into the input box and sign in:
Local Kubernetes cluster with Minikube
Minikube can be added to an existing Docker Desktop install. Minikube can either use the pre-installed hypervisor or use a hypervisor of choice. These instructions use HyperKit which is embedded in Docker Desktop.
If you want to use a different hypervisor then HyperKit make sure that you follow the generic minikube install instructions. Do not forget to install the correct driver for the chosen hypervisor if required. The basic instructions are provided in the minikube install instructions.
Check hypervisor Docker Desktop should have already installed HyperKit. In a terminal run:
hyperkit to confirm. Any response other than
hyperkit: command not found confirms that HyperKit is installed and on the path. If it is not found you can choose a different hypervisor or fix the Docker Desktop install.
- install minikube, you can either use brew or directly via these steps:
curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64
chmod +x minikube
sudo mv minikube /usr/local/bin
- install HyperKit driver (required), you can either use brew or directly via these steps:
curl -LO https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-hyperkit
sudo install -o root -g wheel -m 4755 docker-machine-driver-hyperkit /usr/local/bin/
- update the minikube config to default to the HyperKit install
minikube config set vm-driver hyperkit
- change docker desktop to use minikube for Kubernetes:
Deploy and access the cluster
After the installation is done you can start a new cluster.
- start the minikube cluster:
minikube start --kubernetes-version v1.24.7
- start the minikube dashboard:
minikube dashboard &
When you create images make sure that the build is run after pointing it to the right environment. Without setting the enviromnent minikube might not find the docker images when deploying the scheduler.
- make sure minikube is started
- in the terminal where you wll run the build execute:
eval $(minikube docker-env)
- run the image build from the yunikorn-k8shim repository root:
- deploy the scheduler as per the normal instructions.
Local Kubernetes Cluster with Kind
Kind (Kubernetes in Docker) is a lightweight tool for running lightweight Kubernetes environments. It is very easy to test different Kubernetes versions with kind. You can just select the kind image you want.
If you have go installed, you can run
go install sigs.k8s.io/kind@latest.
Other ways can be found on the Kind website
To use Kind with Kubernetes 1.25, you will need to use email@example.com or greater. The release of kind does allow for particular versions of Kubernetes and you can get that information from the Kind release notes.
To test a new version of Kubernetes, you can pull a corresponding image from kind's repo.
Creating a v1.24.7 Kubernetes Cluster:
kind create cluster --name test --image kindest/node:v1.24.7
Deleting a kind cluster:
kind delete cluster --name test
Loading your images
In order to use a local image, you have to load your images into kind's registry. If you run
make image, you could use the following command to load your kind image. This assumes AMD64 architecture.
The scheduler, web-ui and admission-controller examples are below:
kind load docker-image apache/yunikorn:scheduler-amd64-latest
kind load docker-image apache/yunikorn:web-amd64-latest
kind load docker-image apache/yunikorn:admission-amd64-latest
Debug code locally
Note, this instruction requires you have GoLand IDE for development.
In GoLand, go to yunikorn-k8shim project. Then click "Run" -> "Debug..." -> "Edit Configuration..." to get the pop-up configuration window.
Note, you need to click "+" to create a new profile if the
Go Build option is not available at the first time.
The highlighted fields are the configurations you need to add. These include:
- Run Kind: package
- Package path: point to the path of
- Working directory: point to the path of the
confdirectory, this is where the program loads configuration file from
- Program arguments: specify the arguments to run the program, such as
-kubeConfig=/path/to/.kube/config -interval=1s -clusterId=mycluster -clusterVersion=0.1 -name=yunikorn -policyGroup=queues -logEncoding=console -logLevel=-1. Note, you need to replace
/path/to/.kube/configwith the local path to the kubeconfig file. And if you want to change or add more options, you can run
_output/bin/k8s-yunikorn-scheduler -hto find out.
Once the changes are done, click "Apply", then "Debug". You will need to set proper breakpoints in order to debug the program.
Access remote Kubernetes cluster
This setup assumes you have already installed a remote Kubernetes cluster. For a generic view on how to access a multiple cluster and integrate it follow the accessing multiple clusters documentation from Kubernetes.
Or follow these simplified steps:
- get the Kubernetes
configfile from remote cluster, copy it to the local machine and give it a unique name i.e.
- save the
KUBECONFIGenvironment variable (if set)
- add the new file to the environment variable
- run the command
kubectl config viewto check that both configs can be accessed
- switch context using
kubectl config use-context my-remote-cluster
- confirm that the current context is now switched to the remote cluster config:
kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
docker-for-desktop docker-for-desktop-cluster docker-for-desktop
* my-remote-cluster kubernetes kubernetes-admin
More docs can be found here