Deep Equilibrium Models

Speaker: Shaojie BAI
         Carnegie Mellon University

Title:   "Deep Equilibrium Models"

Date:    Tuesday, 1 March 2022

Time:    10:00am - 11:00am (HKT)

Zoom link:
https://hkust.zoom.us/j/928308079?pwd=MW9wTCtlSDd2MnViZGdNd2oreUpXZz09

Meeting ID:     928 308 079
Passcode:       20212022

Abstract:

Modern artificial intelligence (AI), and the field of deep learning in
particular, has achieved remarkable success on a variety of domains, using
networks that have tens to hundreds of layers and billions of parameters.
But does deep learning actually need to be deep? In my talk, I will
present our recent and ongoing work on Deep Equilibrium (DEQ) Models, an
approach that demonstrates we can achieve most of the benefits of modern
deep learning systems using very shallow models.  But unlike traditional
deep networks, these models need to be defined **implicitly** via finding
fixed points of nonlinear dynamical systems. I will show that these
methods can achieve results on par with the state-of-the-art in domains
spanning large-scale language modeling, image classification, semantic
segmentation, and optical flow, while requiring only O(1) memory and
simplifying architectures substantially; this raises exciting
opportunities to apply performant AI models to more optimization-based,
low-resource and low-latency (i.e., real-time) settings.  I will conclude
by discussing ongoing work and future directions for this class of models
in these areas.


*********************
Biography:

Shaojie Bai is a graduating Ph.D. student in the Machine Learning
Department of Carnegie Mellon University (CMU), advised by J. Zico Kolter.
Shaojie's research focuses on deep learning architectures, with a specific
focus on 1) the scalability and representational capacity of
implicit-depth models, which involves rethinking some of the current "deep
approaches"; and 2) the unification of different model families in deep
sequence modeling.  He was a J.P. Morgan AI Ph.D. fellow, his work has
received multiple spotlight and oral presentations at AI conferences, and
he led a team that won 1st place in a competition on predicting molecular
properties.  Previously, Shaojie received his B.S. in Computer Science and
B.S. in Applied Mathematics from CMU in 2017, where he graduated with
University Honor.