A Survey on Gradient-based Bi-level Optimization and its Applications in Machine Learning

PhD Qualifying Examination


Title: "A Survey on Gradient-based Bi-level Optimization and its Applications 
in Machine Learning"

by

Mr. Hansi YANG


Abstract:

Gradient-based bi-level optimization has emerged as a powerful paradigm in 
the deep learning community, enabling the efficient optimization of 
hyperparameters and extraction of meta-knowledge. This family of algorithms 
has gained significant attention due to its computational efficiency compared 
to classical methods like evolutionary algorithms. This survey provides a 
thorough and comprehensive overview of gradient-based bi-level optimization, 
covering its formal definition, problem structuring, and solution strategies. 
We begin by formally defining gradient-based bi-level optimization and 
outlining the criteria for determining whether a research problem is amenable 
to this approach. We then discuss four bi-level optimization solvers to 
compute the hypergradients for the outer level variable, including explicit 
gradient update, proxy update, implicit function update, and closed-form 
update. We further introduce iterative algorithms to update the inner and 
outer level variables, and review current theoretical results on the 
convergence of these algorithms. We then introduce two important formulations 
for real-world applications of bi-level optimization in machine learning: the 
single-task formulation for optimizing hyperparameters, such as 
regularization parameters and distilled data, and the multi-task formulation 
for extracting meta-knowledge, including model initialization. Finally, we 
summarize the contribution of this survey and outline some potential further 
research questions.


Date:                   Thursday, 19 December 2024

Time:                   10:00am - 12:00noon

Venue:                  Room 5564
                        Lifts 27/28

Committee Members:      Prof. James Kwok (Supervisor)
                        Prof. Raymond Wong (Chairperson)
                        Dr. Shimin Di
                        Dr. Dan Xu