Accelerating Distributed Deep Learning Tasks on Image Datasets

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


PhD Thesis Defence


Title: "Accelerating Distributed Deep Learning Tasks on Image Datasets"

By

Mr. Lipeng WANG


Abstract

Large-scale deep learning tasks usually run on parallel and distributed 
frameworks such as TensorFlow and PyTorch, and take hours to days to obtain 
training results. These frameworks utilize hardware accelerators, especially 
GPUs, to speed up the computation. However, data access and processing in these 
tasks takes a significant amount of time. Therefore, we propose to accelerate 
these tasks by improving their dataset storage and processing. Firstly, we 
develop DIESEL, a scalable dataset storage and caching system that runs between 
a training framework and the underlying distributed file system. The main 
features of DIESEL include metadata snapshot, per-task distributed cache, and 
chunk-based storage and shuffle. Secondly, we optimize a GPU-assisted image 
decoding method for training tasks on image datasets. Furthermore, we introduce 
an online region-of-interest (ROI) method to reduce the data movement cost 
between computer nodes. Our experiments on real-world training tasks show that 
(1) DIESEL halves the data access time and reduces the training time by around 
15%-27%, (2) our optimized image decoding method is 30%-50% faster than 
existing GPU-accelerated image decoding libraries, and (3) our online ROI 
method reduces the data transfer time between DIESEL's caching layer to the 
deep learning framework by around 50%. Overall, our system outperforms existing 
systems by a factor of two to three times on the end-to-end running time of 
deep learning tasks on image datasets.


Date:			Monday, 16 November 2020

Time:			2:30pm - 4:30pm

Zoom Meeting: 
https://hkust.zoom.us/j/93784156069?pwd=bFBmcmZndzhLblpmd3kyNWZ4OFBJdz09

Chairperson:		Prof. Lixin WU (MATH)

Committee Members:	Prof. Qiong LUO (Supervisor)
 			Prof. Kai CHEN
 			Prof. Qifeng CHEN
 			Prof. Wei ZHANG (ECE)
 			Prof. Xiaowen CHU (HKBU)

**** ALL are Welcome ****