Dynamic Hierarchical Mimicking Towards Consistent Optimization Objectives

MPhil Thesis Defence


Title: "Dynamic Hierarchical Mimicking Towards Consistent Optimization 
Objectives"

By

Mr. Duo LI


Abstract

While the depth of modern Convolutional Neural Networks (CNNs) surpasses 
that of the pioneering networks with a significant margin, the traditional 
way of appending supervision only over the final classifier and 
progressively propagating gradient flow upstream remains the training 
mainstay. Seminal Deeply-Supervised Networks (DSN) were proposed to 
alleviate the difficulty of optimization arising from gradient flow 
through a long chain. However, it is still vulnerable to issues including 
interference to the hierarchical representation generation process and 
inconsistent optimization objectives, as illustrated theoretically and 
empirically in this paper. Complementary to previous training strategies, 
we propose Dynamic Hierarchical Mimicking, a generic feature learning 
mechanism, to advance CNN training with enhanced generalization ability. 
Partially inspired by DSN, we fork delicately designed side branches from 
the intermediate layers of a given neural network. Each branch can emerge 
from certain locations of the main branch dynamically, which not only 
retains representation rooted in the backbone network but also generates 
more diverse representations along its own pathway. We go one step further 
to promote multi-level interactions among different branches through an 
optimization formula with probabilistic prediction matching losses, thus 
guaranteeing a more robust optimization process and better representation 
ability. Experiments on both category and instance recognition tasks 
demonstrate the substantial improvements of our proposed method over its 
corresponding counterparts using diverse state-of-the-art CNN 
architectures. Code and models are publicly available at 
https://github.com/d-li14/DHM.


Date:  			Wednesday, 28 July 2021

Time:			3:00pm - 5:00pm

Zoom meeting: 
https://hkust.zoom.us/j/91935373324?pwd=bS9yclpBR1BlYVZtZVpHUFdDaVFYZz09

Committee Members:	Dr. Qifeng Chen (Supervisor)
 			Dr. Dan Xu (Chairperson)
 			Dr. Hao Chen


**** ALL are Welcome ****