Skip to main content

Upcoming Events​

  • KubeCon and CloudNativeCon 2023, Amsterdam
    Scheduling Framework: Batch Extensions with Apache YuniKorn
    https://sched.co/1Hydt (Thursday, April 20th 4.30PM CEST)
    Wilfred Spiegelenburg and Peter Bacsko (Cloudera)
  • KubeCon and CloudNativeCon 2023, Batch + HPC Day event, Amsterdam SLA Aware Batch Scheduling in Apache YuniKorn with Multi-Tenant Preemption
    https://sched.co/1Jo90 (Tuesday, April 18th 3.50PM CEST)
    Craig Condit and Sunil Govindan (Cloudera)

Join us in person or virtual. We wil post links to the recordings of the sessions after they have been made available.

Upcoming Meetup​

None planned at this moment, check for updates later...

Past Meetups​

4:30pm - 6:00pm, PST, 20 October 2022

Presentations and recording links located below

We are hosting an Apache YuniKorn meetup on Thursday, 20 October 2022. Come and join us! This is the first hybrid on-line and in-person meetup for Apache YuniKorn.

A lot has changed since our last meetup a little over 6 months ago. We since then have graduated to an Apache TLP, released v1.0.0 and two minor releases as a follow-up. We had multiple presentations at: KubeCon Europe 2022, VISA Meetup and ApacheCON 2022. This is just a partial list of all that has happened and is going to happen.

To add this event to your Google calendar [click here]. The invite includes a link for joining us virtually [using zoom webinar].

We have the following agenda lined up:

  1. YuniKorn @ Pinterest
    Speaker : Rainie, Pinterest
    Abstract: Spark on YARN is used for production workloads. Using Apache YuniKorn we try to move from YARN to Kubernetes. A path of testing, comparing and changing at multiple levels.
  2. ApacheCON and YuniKorn : Feedback
    Speaker : Wilfred, Cloudera
    Abstract: As a community we delivered two talks at ApacheCON NA. How did we do and what was the feedback on our two talks. Cloud native also had its own track: some observations from other talks.
  3. YuniKorn roadmap
    Speaker : Wilfred / Craig, Cloudera
    Abstract: For YuniKorn 1.2 we have a roadmap defined around User Quotas, Application Priority and a K8s upgrade. Is that all we want and need. Do we want to add other things, and what do we want to do after this release?

There will also plenty of time to discuss community related things that are not on the agenda.

4:30pm - 5:30pm, PST, Feb 16, 2022

We are hosting an Apache YuniKorn (Incubating) meetup on Wednesday February 16. Come join us! Since our first meetup held in November last year, lots of new development has happened in the community. We are excited to have three talks lined up! Add this event to your calendar: [calendar]

Erica Lin, Luna Xu, and Sean Gorsky from Sync Computing will present their work on Spark autotuner and orchestrator. Description: Efficiently managing the infrastructure and schedules of thousands of data pipelines in multi-tenant and heterogeneous environments is a daunting task. Poor application tuning, resource allocation, and scheduling can lead to exorbitant costs on the cloud, sluggish performance, and failed jobs due to the intractable infrastructure search space. Addressing each of these codependent issues separately also often doesn't lead to the best results overall. Sync Computing will share how their autotuner and orchestrator addresses these issues jointly as a single optimization problem. As a result, the solution offers globally optimized Spark configurations, resource provisioning, and job scheduling with configurable optimization goals, enabling a seamless user experience of running DAG workflows in the cloud. We experimentally demonstrate on AWS up to 77% cost savings or 45% performance acceleration of a Spark and Airflow DAG. Simulations of our solution on a multi-day Alibaba cloud trace resulted in a 57% reduction in total DAG completion time for their batch workloads. This work could be complementary to Yunikorn and we would like to discuss potential integration strategies with the community.

Craig Condit from Cloudera will talk about the latest YuniKorn release v0.12.2 and offer a glimpse of what the upcoming v1.0 release has in store. Description: v0.12.2 release offers a host of new features and bug fixes. Specifically, the admission controller implementation has been refactored to enable the YuniKorn scheduler to be seamless maintained and upgraded. In the upcoming v1.0 release, we will witness a major milestone of the YuniKorn project. An exciting feature is a new mode of deployment based on the Kubernetes Scheduling Framework, which offers better compatibility with the default scheduler and makes it easier for new users to try out YuniKorn.

Tingyao Huang and Yuteng Chen are both YuniKorn committers. They have done extensive performance benchmarking using the latest YuniKorn codebase. Their results showed convincing throughput advantages of YuniKorn compared to the default scheduler.

4:30pm - 6:00pm, PST, Nov 18, 2021

Wilfred Spiegelenburg presented a session to introduce the latest status in the YuniKorn community.

Abstract: Apache YuniKorn (Incubating) has released v0.11 earlier this year with a number of new features and improvements like Gang scheduling, REST API enhancements and Kubernetes 1.19 support. In a month, we are planning the major v1.0.0 release with Kubernetes 1.20 & 1.21 support, improved node sorting and numerous small fixes & enhancements! In this meetup, we will deep dive into the implementation of Gang scheduling behind the use of temporary placeholder pods on Kubernetes, significant performance improvement with simplified scheduler core code and a new node sorting algorithm. Please find all the topics for this meetup here.

Past Conference & Meetup Recordings​

Demo Videos​

Please subscribe to YuniKorn Youtube Channel to get notification about new demos!

If you are a YuniKorn evangelist, and you have public conference talks, demo recording that related to YuniKorn. Please submit a PR to extend this list!