- KubeCon and CloudNativeCon 2023, Amsterdam
Scheduling Framework: Batch Extensions with Apache YuniKorn
https://sched.co/1Hydt (Thursday, April 20th 4.30PM CEST)
Wilfred Spiegelenburg and Peter Bacsko (Cloudera)
- KubeCon and CloudNativeCon 2023, Batch + HPC Day event, Amsterdam
SLA Aware Batch Scheduling in Apache YuniKorn with Multi-Tenant Preemption
https://sched.co/1Jo90 (Tuesday, April 18th 3.50PM CEST)
Craig Condit and Sunil Govindan (Cloudera)
Join us in person or virtual. We wil post links to the recordings of the sessions after they have been made available.
None planned at this moment, check for updates later...
4:30pm - 6:00pm, PST, 20 October 2022
Presentations and recording links located below
We are hosting an Apache YuniKorn meetup on Thursday, 20 October 2022. Come and join us! This is the first hybrid on-line and in-person meetup for Apache YuniKorn.
A lot has changed since our last meetup a little over 6 months ago. We since then have graduated to an Apache TLP, released v1.0.0 and two minor releases as a follow-up. We had multiple presentations at: KubeCon Europe 2022, VISA Meetup and ApacheCON 2022. This is just a partial list of all that has happened and is going to happen.
To add this event to your Google calendar [click here]. The invite includes a link for joining us virtually [using zoom webinar].
We have the following agenda lined up:
- YuniKorn @ Pinterest
Speaker : Rainie, Pinterest
Abstract: Spark on YARN is used for production workloads. Using Apache YuniKorn we try to move from YARN to Kubernetes. A path of testing, comparing and changing at multiple levels.
- ApacheCON and YuniKorn : Feedback
Speaker : Wilfred, Cloudera
Abstract: As a community we delivered two talks at ApacheCON NA. How did we do and what was the feedback on our two talks. Cloud native also had its own track: some observations from other talks.
- YuniKorn roadmap
Speaker : Wilfred / Craig, Cloudera
Abstract: For YuniKorn 1.2 we have a roadmap defined around User Quotas, Application Priority and a K8s upgrade. Is that all we want and need. Do we want to add other things, and what do we want to do after this release?
There will also plenty of time to discuss community related things that are not on the agenda.
4:30pm - 5:30pm, PST, Feb 16, 2022
We are hosting an Apache YuniKorn (Incubating) meetup on Wednesday February 16. Come join us! Since our first meetup held in November last year, lots of new development has happened in the community. We are excited to have three talks lined up! Add this event to your calendar: [calendar]
Erica Lin, Luna Xu, and Sean Gorsky from Sync Computing will present their work on Spark autotuner and orchestrator. Description: Efficiently managing the infrastructure and schedules of thousands of data pipelines in multi-tenant and heterogeneous environments is a daunting task. Poor application tuning, resource allocation, and scheduling can lead to exorbitant costs on the cloud, sluggish performance, and failed jobs due to the intractable infrastructure search space. Addressing each of these codependent issues separately also often doesn't lead to the best results overall. Sync Computing will share how their autotuner and orchestrator addresses these issues jointly as a single optimization problem. As a result, the solution offers globally optimized Spark configurations, resource provisioning, and job scheduling with configurable optimization goals, enabling a seamless user experience of running DAG workflows in the cloud. We experimentally demonstrate on AWS up to 77% cost savings or 45% performance acceleration of a Spark and Airflow DAG. Simulations of our solution on a multi-day Alibaba cloud trace resulted in a 57% reduction in total DAG completion time for their batch workloads. This work could be complementary to Yunikorn and we would like to discuss potential integration strategies with the community.
Craig Condit from Cloudera will talk about the latest YuniKorn release v0.12.2 and offer a glimpse of what the upcoming v1.0 release has in store. Description: v0.12.2 release offers a host of new features and bug fixes. Specifically, the admission controller implementation has been refactored to enable the YuniKorn scheduler to be seamless maintained and upgraded. In the upcoming v1.0 release, we will witness a major milestone of the YuniKorn project. An exciting feature is a new mode of deployment based on the Kubernetes Scheduling Framework, which offers better compatibility with the default scheduler and makes it easier for new users to try out YuniKorn.
Tingyao Huang and Yuteng Chen are both YuniKorn committers. They have done extensive performance benchmarking using the latest YuniKorn codebase. Their results showed convincing throughput advantages of YuniKorn compared to the default scheduler.
4:30pm - 6:00pm, PST, Nov 18, 2021
Wilfred Spiegelenburg presented a session to introduce the latest status in the YuniKorn community.
Abstract: Apache YuniKorn (Incubating) has released v0.11 earlier this year with a number of new features and improvements like Gang scheduling, REST API enhancements and Kubernetes 1.19 support. In a month, we are planning the major v1.0.0 release with Kubernetes 1.20 & 1.21 support, improved node sorting and numerous small fixes & enhancements! In this meetup, we will deep dive into the implementation of Gang scheduling behind the use of temporary placeholder pods on Kubernetes, significant performance improvement with simplified scheduler core code and a new node sorting algorithm. Please find all the topics for this meetup here.
Past Conference & Meetup Recordings
- KubeCon North America 2022, Batch & HPC day:
- Beyond Experimental: Spark on Kubernetes 👥 Weiwei Yang, Bowen Li
- Panel discussion: Batch scheduling on Kubernetes 👥 Wilfred Spiegelenburg, Et al.
- Apache YuniKorn Meetup: October 2022 | Pinterest presentation | Cloudera presentation 👥 Wilfred Spiegelenburg, Rainie Li, Craig Condit
- ApacheCon North America 2022: Apache YuniKorn Enhanced Scheduling in the Cloud 👥 Wilfred Spiegelenburg, Manikandan Ramaraj
- VISA Data Meetup: Apache YuniKorn scheduling Apache Spark on Kubernetes 👥 Wilfred Spiegelenburg
- KubeCon Europe 2022, Batch & HPC day: Apache YuniKorn A Kubernetes Scheduler Plugin for Batch Workloads 👥 Wilfred Spiegelenburg, Craig Condit
- ApacheCon 2021: Next Level Spark on Kubernetes with Apache YuniKorn (Incubating) 👥 Weiwei Yang, Chaoran Yu
- ApacheCon Asia 2021: State Of The Union With Apache Yunikorn (Incubating) - Cloud Native Scheduler For Big Data Usecases 👥 Julia Kinga Marton, Sunil Govindan
- Future of Data Meetup: Apache YuniKorn: Cloud-native Resource Scheduling 👥 Wangda Tan, Wilfred Spiegelenburg
- ApacheCon 2020: Integrate Apache Flink with Cloud Native Ecosystem 👥 Yang Wang, Tao Yang
- Spark+AI Summit 2020: Cloud-Native Spark Scheduling with YuniKorn Scheduler 👥 Gao Li, Weiwei Yang
- Flink Forward 2020: Energize multi-tenant Flink on K8s with YuniKorn 👥 Weiwei Yang, Wilfred Spiegelenburg
Please subscribe to YuniKorn Youtube Channel to get notification about new demos!
- Running YuniKorn on Kubernetes - a 12 minutes Hello-world demo
- YuniKorn configuration hot-refresh introduction
- Yunikorn scheduling and volumes on Kubernetes
- Yunikorn placement rules for applications
- Federated K8s compute pools with YuniKorn
If you are a YuniKorn evangelist, and you have public conference talks, demo recording that related to YuniKorn. Please submit a PR to extend this list!