Acceleration Techniques and Systems for Training Deep Neural Networks

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


MPhil Thesis Defence


Title: "Acceleration Techniques and Systems for Training Deep Neural Networks"

By

Mr. Rui PAN


Abstract:

The large training cost of modern deep neural networks, such as transformers, 
has been a major roadblock in scaling up model and dataset sizes, slowing down 
the progress of developing more intelligent systems. To accelerate the 
optimization process of modern transformer models, efforts have been put into 
the different aspects of the training procedure, including optimization-level, 
dataset-level, and system-level improvements. In this thesis, we briefly review 
developments arising from those different research directions, providing an 
overall picture of current acceleration techniques.

On top of that, we delve deeper into the optimization-level improvement with 
prompt learning, a branch of approaches that distinguishes itself from others 
in enabling black- box learning of large-scale models. We introduce a novel 
combination of metaheuristics and black-box prompt learning, bringing forth a 
series of new prompt-tuning algorithms. Within our paradigm, we develop and 
examine six typical methods: hill climbing, simulated annealing, genetic 
algorithms with/without crossover, tabu search, and harmony search, 
demonstrating their effectiveness under white-box and black-box prompt learning 
settings. In particular, we show that these methods can be employed to discover 
more human-understandable prompts that were previously unknown in both 
reasoning and image generation tasks, opening the door to a cornucopia of 
possibilities in prompt optimization. This enables the optimization of 
black-box models to be more efficient, allowing diverse task-tailored models to 
be developed with minimum computational resources.


Date:                   Monday, 22 July 2024

Time:                   10:00am - 12:00noon

Venue:                  Room 3494
                        Lifts 25/26

Chairman:               Prof. Xiaofang ZHOU

Committee Members:      Dr. Yangqiu SONG (Supervisor)
                        Prof. Dit-Yan YEUNG