D-Adaptation for learning rate free optimization

Speaker: Dr. Aaron Defazio
         Meta AI

Title:  "D-Adaptation for learning rate free optimization"

Date:   Tuesday, 14 March 2023

Time:   2:00pm - 3:00pm

Venue:  Room 2503 (via lift 25/26), HKUST

Abstract:

D-Adaptation is a new hyper-parameter free method which asymptotically
achieves the optimal rate of convergence for minimizing convex Lipschitz
functions, with no back-tracking, line searches or other additional
computation per step. Our approach is the first parameter-free method for
this class without additional multiplicative log factors in the
convergence rate. We present extensive experiments for SGD and Adam
variants of our method, where the method automatically matches hand-tuned
learning rates across more than a dozen diverse machine learning problems,
including large-scale vision and language problems.


*****************
Biography:

My research interests cover the intersection of machine learning and
applied mathematics. I develop techniques grounded in sound mathematical
theory for solving real world problems. My work for Meta Platforms, Inc.
includes novel algorithms for MR imaging reconstruction and automated
theorem proving. I also work on techniques for faster and more reliable
model training, including new optimization algorithms and initialization
techniques for training production models, including feed ranking and
recommendation systems. I developed the SAGA algorithm, which is now
actively used in industry and taught in graduate courses.