A SURVEY ON LEARNING RATE SCHEDULES FOR TRAINING DEEP LEARNING MODELS

PhD Qualifying Examination


Title: "A SURVEY ON LEARNING RATE SCHEDULES FOR TRAINING DEEP LEARNING MODELS"

by

Mr. Rui PAN


Abstract:

Learning rate schedule is a key ingredient in optimizing deep neural networks 
with gradient-based methods. In practice, deep learning models that achieve 
state-of-the-art performance normally require selection and tuning of different 
types of learning rate schedules. This survey aims to provide a comprehensive 
review of common learning rate schedules used in practice, along with their 
last iterate convergence properties for stochastic gradient descent (SGD) on 
convex or special non-convex objectives. We then present a novel perspective to 
understand the possible reason behind this diversity of effective schedules, 
followed by a framework that produces task-dependent schedules with strong 
theoretical guarantees on strongly convex least square regressions. Other 
relevant optimization techniques, e.g. Newton’s method, adaptive gradients are 
also discussed.


Date:  			Thursday, 28 July 2022

Time:                  	9:00am to 11:00am

Zoom Meeting:
https://hkust.zoom.us/j/91624044687?pwd=SnF1YityeHhwbGV3bkswWllyQmxYUT09

Committee Members:	Prof. Tong Zhang (Supervisor)
 			Prof. Kai Chen (Chairperson)
 			Prof. Raymond Wong
 			Prof. Dit-Yan Yeung


**** ALL are Welcome ****