Optimization of Spark for divide and conquer algorithms

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering

Final Year Thesis Oral Defense

Title: "Optimization of Spark for divide and conquer algorithms"

by

CHEN Liangwei

Abstract:

Spark is the thriving distributed processing platform. Even though there 
are many optimizations on Spark for iterative algorithms, few work has 
been done to provide optimization of Spark for divide and conquer 
algorithms. In this work we conduct experiments the influence of 
computation resources, distributive extent, and parameter setting for 
general divide and conquer algorithms on Spark. By analyzing the 
statistics we propose a scheme to set Spark configuration for optimizing 
execution of divide and conquer algorithms.


Date            : 25 April 2018 (Wednesday)

Time            : 11:00 - 11:40

Venue           : Room 2304 (via lifts 17/18), HKUST

Advisor         : Dr. YI Ke

2nd Reader      : Dr. WANG Wei