Skip to main content

Release Announcement v0.9.0

It gives us great pleasure to announce that the Apache YuniKorn (Incubating) community has voted to release 0.9.0. This is the 2nd release as an Apache Incubator project.

Apache YuniKorn (Incubating) is a standalone resource scheduler, designed for managing, and scheduling Big Data workloads on the container orchestration framework for on-prem and on-cloud use cases.
This release has further improved the capabilities of resource scheduling and management on Kubernetes, along with some optimizations for running Apache Spark, Apache Flink workloads.

Overview​

The Apache YuniKorn (Incubating) community has fixed over 130 JIRAs in this release: YuniKorn jiras fixed in 0.9.0

Release manager: Wilfred Spiegelenburg

Release date: 2020-08-28

Highlights​

Resource Quota Management​

This version YuniKorn provides a seamless way to manage resource quota for a Kubernetes cluster, it can work as an alternative to the namespace resource quota.
There are 2 main advantages of using this feature comparing to the namespace resource quota:

  1. The namespace resource quota is counting resources at the admission phase, irrespective of whether the pod is using the resources or not. This can lead up to issues that the namespace resources could not be efficiently used.
  2. The namespace resource quota is flat, it doesn't support hierarchy resource quota management.

The resource quota admission controller rejects the pods if it could cause the namespace to exceed its quota, this increases the complexity of the client side code.
By using the resource quota management provided by YuniKorn, it is more efficient, seamlessly setup and it provides the job queue to handle common scheduling ordering requirements.

Job Ordering Policy: StateAware (optimized FIFO)​

The StateAware app sorting policy orders jobs in a queue in FIFO order, and schedules them one by one on conditions. The condition is to wait for the application to enter a runnable state. This avoids the common race condition while submitting lots of batch jobs, e.g Spark, to a single namespace (or cluster). By enforcing the certain ordering of jobs, it also improves the scheduling of jobs to be more predictable. More explanation of this feature can be found in doc here.

Improvements for Auto-scaling​

In this release, YuniKorn has been tested heavily to work nicely with the Kubernetes cluster-autoscaler. It brings the maximum elasticity to the Kubernetes cluster by working efficiently with the cluster-autoscaler. Some bugs are fixed and some improvements are done in this release.

Event Cache System​

In this release, an efficient event cache system is added into the scheduler. This system caches some key scheduling events in a memory store and publishes them to the Kubernetes event system when needed. More scheduling events are visible directly from Kubernetes by using kubectl command. This helps to improve the usability and debuggability a lot.

More Comprehensive Web UI​

YuniKorn UI provides a better centralized view for resource management. A new "nodes" page has been added to the UI, to display the detailed nodes info in the cluster.
The apps page has been enhanced, it now provides a search box to search apps by queue name or application ID.

Community​

The Apache YuniKorn community is pleased to welcome a new committer Gao Li. We would love to see more committers joining the community and help to shape the project. In the past few months, we continue to have bi-weekly community meetings in 2 different time zones. Thanks for all the community folks who joined these meetings and provided feedback and valuable ideas.

Thanks to the following contributors who directly contributed code to the project: Adam Antal, Akhil Puthenveettil Balan, Ayub Khan Pathan, Kinga Marton, Manikandan R, Sunil Govindan, Tao Yang, Tingyao Huang, Tim Rots, Wangda Tan, Wanqiang Ji, Weiwei Yang, Wilfred Spiegelenburg.