Provable Convergence of Generative Learning via Smoothed Functional Gradient Optimization

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


MPhil Thesis Defence


Title: "Provable Convergence of Generative Learning via Smoothed Functional
Gradient Optimization"

By

Mr. Jingwei ZHANG


Abstract:

Generative learning aims to learn the underlying data distributions based on
finite observations. This can be achieved by minimizing some divergence between
the generated distribution and the real data distribution. For example, the
vanilla GAN [2] minimizes the Jensen-Shannon divergence between the generator
distribution and the real data distribution under the conditions of an optimal
discriminator. While the formulation of divergence minimization via gradient
flow has been extensively studied both theoretically and empirically, the
convergence analysis of the gradient flow, either qualitatively or
quantitively, is quite limited. Analyzing the convergence is a technically
challenging task due to the unbounded and nonlinear nature of the partial
differential equation of McKean-Vlasov type that describes the dynamics of the
gradient flow. In this thesis, we consider the regularized smoothed
KL-divergence as the generative learning objective and study the convergence
property of its gradient flow endowed with 2-Wasserstein metric. Our
contributions lie in three folds: (1). We prove the first asymptotic global
convergence of the KL gradient flow based on the Gaussian-Poincare inequality
established from the quadratic bound of the smoothed functional gradient; (2).
Under more refined analysis, we prove the first quantitive linear convergence
of the smoothed KL gradient flow to the global optima by estabilishing uniform
Log-Sobolev inequalities for the proximal Gibbs distributions corresponding to
the generator; (3). We also consider different discretizations of the gradient
flow and approximate the functional gradient by neural networks via score
matching. This approximation yields implementable algorithms that we called
smoothed functional gradient optimization (SFGO) for generative learning and
learns a generator by layer-wise training and aggregation of some simple neural
networks. We finally conduct numerical experiments to validate the
effectiveness of SFGO.


Date:                   Wednesday, 29 November 2023

Time:                   10:30am - 12:30pm

Venue:                  Room 4472
                        lifts 25/26

Committee Members:      Prof. Tong Zhang (Supervisor)
                        Prof. James Kwok (Chairperson)
                        Dr. Can Yang (MATH)


**** ALL are Welcome ****