Architecture

Apache YuniKorn (Incubating) is a light-weight, universal resource scheduler for container orchestrator systems. It is created to achieve fine-grained resource sharing for various workloads efficiently on a large scale, multi-tenant, and cloud-native environment. YuniKorn brings a unified, cross-platform, scheduling experience for mixed workloads that consist of stateless batch workloads and stateful services.

YuniKorn now supports K8s and can be deployed as a custom K8s scheduler. YuniKorn's architecture design also allows adding different shim layer and adopt to different ResourceManager implementation including Apache Hadoop YARN, or any other systems.

Architecture

Following chart illustrates the high-level architecture of YuniKorn.

Components

Scheduler interface

Scheduler interface is an abstract layer which resource management platform (like YARN/K8s) will speak with, via API like GRPC/programing language bindings.

Scheduler core

Scheduler core encapsulates all scheduling algorithms, it collects resources from underneath resource management platforms (like YARN/K8s), and is responsible for container allocation requests. It makes the decision where is the best spot for each request and then sends response allocations to the resource management platform. Scheduler core is agnostic about underneath platforms, all the communications are through the scheduler interface. Please read more about the design of schedule core here.

Kubernetes shim

The YuniKorn Kubernetes shim is responsible for talking to Kubernetes, it is responsible for translating the Kubernetes cluster resources, and resource requests via scheduler interface and send them to the scheduler core. And when a scheduler decision is made, it is responsible for binding the pod to the specific node. All the communication between the shim and the scheduler core is through the scheduler interface. Please read more about the design of the Kubernetes shim here