Release Announcement v0.8.0

This is the first release of Apache YuniKorn (Incubating).

Overview

The Apache YuniKorn (Incubating) community has fixed 60 JIRAs in this release: YuniKorn jiras fixed in 0.8.0

Release manager: Weiwei Yang

Release date: 2020-05-04

Highlights

Queues and placement rules

With this new dynamic queue management feature, users can set up placement rules to delegate queue management.
This enables use cases like mapping a K8 namespace to a YuniKorn queue without the need to pre-create all queues.
Queues are created based on placement rules when required for running an application. Dynamic queues are deleted when no longer required.

Pluggable Application Management

A pluggable app management framework is being added to this release, with this feature, YuniKorn can be easily integrated with 3rd party K8s operators, such as Spark-k8s-operator, Flink-k8s-operator, etc.

Resource Reservation

Reservation is the methodology used to avoid large/picky resource requests being starved in the batch scheduling scenario.

One node is reserved for a specific request from an application, effectively removing it from the available node list for other applications.

Pluggable Node Sorting Policies

There are 2 builtin node sorting policies (FAIR and BinPacking) available in YuniKorn with regards to allocation distribution.

The node sorting policies are pluggable, users can implement their own policies and plug that into the scheduler.

Improved Scheduling Performance

When running Big Data batch workloads, e.g Spark, on K8s, scheduler throughput becomes one of the main concerns. In YuniKorn, there are lots of optimizations to improve the performance, such as a fully async event-driven system and low-latency sorting policies.

Based on our test using Kubemark, YuniKorn is 2x+ faster than the default K8s scheduler.

Community

Since the birth of the YuniKorn project, the feedback has been well-received. We are thankful to have such a diverse community where we have members from Microsoft, LinkedIn, Alibaba, Apple, Tencent, Nvidia, and Cloudera, to help and guide the project’s growth. Many thanks to everyone who contributed to the release, and everyone in the Apache YuniKorn (Incubating) community!

This release is a result of direct and indirect efforts from many. The following who contributed directly by submitting patches: Adam Antal, Akhil Puthenveettil Balan, Bhuvan Mysore, Jetly Jaimin, Kinga Marton, Sunil Govindan, Tao Yang, Wangda Tan, Wanqiang Ji, Weiwei Yang, Wilfred Spiegelenburg, Vinod Kumar Vavilapalli.

Thanks to our mentors Felix Cheung, Holden Karau, Jason Lowe, Junping Du for helping us to set up repositories, permissions, and guide us on how to do the development and release the Apache way.