Skip to main content

Release Announcement v0.11.0

It gives us great pleasure to announce that the Apache YuniKorn (Incubating) community has voted to release 0.11.0. Apache YuniKorn (Incubating) is a standalone resource scheduler, designed for managing, and scheduling Big Data workloads on the container orchestration framework for on-prem and on-cloud use cases. This release has a couple of REST API, UI usability and Gang scheduling improvements.

Overview

The Apache YuniKorn (Incubating) community has fixed over 110 JIRAs in this release: YuniKorn jiras resolved in 0.11.0

Release manager: Kinga Marton

Release date: 2021-08-18

Highlights

Supported Kubernetes Versions

In this release supported Kubernetes versions have been updated to 1.17.x, 1.18.x and 1.19.x. (Earlier versions support 1.16.x, 1.17.x and 1.18.x.) YuniKorn support matrix primarily supports 3 major Kubernetes versions.

Gang Scheduling enhancements

The community has been made further improvements to the Gang Scheduling feature. Gang scheduling style now can be configured per app, soft or hard. This allows the users to customize the behavior how to handle the reservation timeout. There are also lots of enhancements to improve the lifecycle management of the placeholder pods, make the cleanup more robust and reliable. More detail can be found in YUNIKORN-553.

Spark operator integration

By leveraging the YuniKorn's app management framework, now it is better integrated with Spark operator. YuniKorn watches spark CRD resources and reacts when necessary. It is totally transparent to use YuniKorn with Spark operator on K8s, no extra configuration needed. This feature is ready for production use. More detail can be found in YUNIKORN-558.

Scheduler Metrics enhancements

YuniKorn exposes core scheduling metrics via Prometheus, in this release, some important metrics such as allocating_latency_seconds, long_pending_apps have been added to the system, based on which the cluster admins can create more alerts to monitor system healthy. There are more enhancements have been made to the existing metrics, such as apps/queues, to make them more readable, and better organized. Please find more details in YUNIKORN-3.

User/Group Resolution

With the work done in YUNIKORN-649, now users can define how to retrieve user/group identity for each app, and pod. This is done via a configurable environment variable USER_LABEL_KEY, so users can specify which label will be used for YuniKorn to retrieve the user identity. The document can be found here.

Web UI and REST API usability enhancements

The community has made some usability improvements for the web UI to provide better user experience. The UI has been simplified to display the useful data in the apps/nodes page, some usability bugs have been fixed. With this release, some new REST API endpoints were introduced to make it possible to fetch per partition/queue information. You can find a complete list of endpoints here.

Community

The Apache YuniKorn community is pleased to welcome new committer Manikandan R and new PPMC member, Kinga Marton. We would love to see more committers joining the community and help to shape the project. In the past few months, we continue to have bi-weekly community meetings in 2 different time zones, ad-hoc meetings, offline channel discussions. The community will continue to be open.