Multi-Resource Fair Sharing with Constraints in Heterogeneous Clusters

MPhil Thesis Defence


Title: "Multi-Resource Fair Sharing with Constraints in Heterogeneous Clusters"

By

Mr. Xiandong QI


Abstract

Today's production clusters host a variety of jobs with diverse resource 
demands, and fair allocation of the cluster resources is critical for 
performance isolation among those jobs. Nonetheless, existing resource 
allocation policies implicitly assume that a job can run at exactly the same 
efficiency with any unit of its usable resources, which usually does not hold 
in practice. In fact, we find in many scenarios that a job can have different 
preferences for different resources. For example, a job can run much faster in 
those machines directly storing its input data, although it can still run in 
other ones after paying a performance penalty. Such heterogeneous resource 
preferences are called by us as soft-constraints, and it is still unclear how 
to define and achieve fairness in the presence of soft-constraints. In this 
work, we propose MTTC, a proactive exchange-based sharing policy for fair 
allocation with soft constraints, and show that it is the only policy 
satisfying all the typical fairness criteria, including an important property 
that prevents selfish users from lying to benefit themselves. Furthermore, we 
approximate the MTTC allocation by an online preference-aware scheduler called 
FSC, and have integrated the FSC prototype into Apache YARN. The effectiveness 
of FSC is confirmed with both testbed experiments in a 65-node Amazon EMR 
cluster and trace-driven simulations. Particularly, the simulation results 
suggest that FSC can reduce the average job completion time by over 54%.


Date:			Wednesday, 12 December 2018

Time:			2:00pm - 4:00pm

Venue:			Room 1511
 			Lifts 27/28

Committee Members:	Dr. Wei Wang (Supervisor)
 			Dr. Kai Chen (Chairperson)
 			Prof. Bo Li


**** ALL are Welcome ****