Build and Run
YuniKorn always works with a container orchestrator system. Currently, a Kubernetes shim (yunikorn-k8shim) is provided which provides a drop-in scheduler for the Kubernetes platform. This document describes how to setup and use a local development environment.
Dev Environment setup
Read the Dev Environment Setup guide first to setup Docker and Kubernetes development environment.
Build YuniKorn
Prerequisites:
- Golang: check the
.go_version
file in the root of the repositories for the version Yunikorn requires. The minimum version can change per release branch. Using earlier Go versions will cause compilation issues.
You can build the scheduler for Kubernetes from the yunikorn-k8shim project. The build procedure will build all components into a single executable that can be deployed and running on Kubernetes.
Start the integrated build process by pulling the yunikorn-k8shim
repository:
mkdir $HOME/yunikorn/
cd $HOME/yunikorn/
git clone https://github.com/apache/yunikorn-k8shim.git
At this point you have an environment that will allow you to build an integrated image for the YuniKorn scheduler.
Build Docker images
Building the Docker images can be triggered by following command:
make image
This will generate images for the scheduler, scheduler plugin, and admission controller.
The images created can be deployed directly on Kubernetes.
Some sample deployments that can be used are found under the
deployments/scheduler
directory of the yunikorn-k8shim
repository. Alternatively, the Helm charts
located within the helm-charts
directory of the yunikorn-release
repository may be used. These match what is used
for release builds.
The configuration of YuniKorn can be customized via a ConfigMap as explained in the scheduler configuration deployment document.
The make image
build command will first build the integrated executables and
then create the docker images. If you want to use pre-built images based on an
offical release, please check the Docker Hub repo.
The default image tags are not suitable for deployments to a private
repository as these would attempt to push to Docker Hub without proper
credentials. You must update the REGISTRY
variable in the Makefile
to
push to an accessible repository. When you update the image tag be aware that
the deployment examples given will also need to be updated to reflect the same
change.
Inspect Docker images
The Docker images built from previous step have embedded some important build
info in the image metadata. You can retrieve this information with docker
inspect
command:
docker inspect apache/yunikorn:scheduler-amd64-latest
docker inspect apache/yunikorn:scheduler-plugin-amd64-latest
docker inspect apache/yunikorn:admission-controller-amd64-latest
The amd64
tag is dependent on your host architecture (i.e. for Intel it would
be amd64
and for Mac M1, it would be arm64
).
This info includes git revisions (last commit SHA) for each component, to help
you understand which version of the source code was shipped by this image. They
are listed as docker image labels
, such as
"Labels": {
"BuildTimeStamp": "2019-07-16T23:08:06+0800",
"Version": "0.1",
"yunikorn-core-revision": "dca66c7e5a9e",
"yunikorn-k8shim-revision": "bed60f720b28",
"yunikorn-scheduler-interface-revision": "3df392eded1f"
}
Dependencies
The dependencies in the projects are managed using go modules.
If you want to modify one of the projects locally and build with your local
dependencies you will need to change the module file. Changing dependencies
requires using go.mod
replace
directives as explained in the
Update dependencies section.
The YuniKorn project has four code repositories:
- yunikorn-scheduler-interface (protobuf interface between core and shim)
- yunikorn-core (core scheduler logic)
- yunikorn-k8shim (Kubernetes-specific shim)
- yunikorn-web (YuniKorn Web UI)
Each of these dependencies is a Go module and there are dependencies between them. During the development cycle it can be required to break the dependency on the committed version from github. This requires making changes in the module file to allow loading a local copy or a forked copy from a different repository.
Additionally, there are two additional auxiliary repositories:
- yunikorn-release (release management scripts and official Helm charts)
- yunikorn-site (source of the yunikorn.apache.org web site)
Affected repositories
The following dependencies exist between the repositories:
Repository | Depends on |
---|---|
yunikorn-core | yunikorn-scheduler-interface |
yunikorn-k8shim | yunikorn-scheduler-interface, yunikorn-core |
yunikorn-scheduler-interface | none |
yunikorn-web | none |
The yunikorn-web
repository has no direct go dependency on the other
repositories. However any change to the yunikorn-core
web services can affect
the web interface.
Making local changes
To make sure that the local changes will not break other parts of the build you should run:
- A full build
make
(build target depends on the repository) - A full unit test run
make test
Any test failures should be fixed before proceeding.
Updating dependencies
The simplest way is to use the replace
directive in the module file.
The replace
directive allows you to override the import path with a new
(local) path. There is no need to change any of the imports in the source code.
The change must be made in the go.mod
file of the repository that has the
dependency.
Using replace
to use of a forked dependency, such as:
replace github.com/apache/yunikorn-core => example.com/some/forked-yunikorn
There is no requirement to fork and create a new repository. If you do not have a repository you can use a local checked out copy too.
Using replace
to use of a local directory as a dependency:
replace github.com/apache/yunikorn-core => /User/example/local/checked-out-yunikorn
For the same dependency using a relative path:
replace github.com/apache/yunikorn-core => ../checked-out-yunikorn
Note: if the replace
directive is using a local filesystem path, then the target
must have a go.mod
file at that location.
Further details can be found on the Go Wiki: When should I use the 'replace' directive?
Build the Web UI
Example deployments reference the
YuniKorn Web UI. The yunikorn-web
project has specific requirements for the build. Follow the steps in the
README to prepare
a development environment and build the Web UI. However, the scheduler is fully
functional without the Web UI.
Run YuniKorn locally
When you have a local development environment setup you can run the scheduler in your local Kubernetes environment. This has been tested in a desktop enviornment with Docker Desktop, Minikube, and Kind. See the Dev Environment Setup guide for further details.
To run a local instance of the scheduler:
make run
This will launch a local scheduler and connect to the Kubernetes cluster
referenced in your KUBECONFIG
or $HOME/.kube/config
.
To run YuniKorn in Kubernetes scheduler plugin mode instead, execute:
make run_plugin
You can also use the same approach to run the scheduler locally but connecting
to a remote kubernetes cluster, as long as the $HOME/.kube/config
file
is pointing to that remote cluster.
Run end-to-end tests
In addition to the unit tests for each project, YuniKorn contains many e2e
(end-to-end) tests in the yunikorn-k8shim
repository which validate
functionaliy of the scheduler on a functioning Kubernetes cluster.
How to run the tests locally is described here.