Challenges and Opportunities in Deep Learning

Speaker:        Dr. Tie-Yan Liu
                Principal Research Manager
                MSRA

Title:          "Challenges and Opportunities in Deep Learning"

Date:           Friday, 2 December 2016

Time:           2:00pm - 3:00pm

Venue:          Lecture Theatre F (near lifts 25/26), HKUST

Abstract:

The success of deep learning could be attributed to the availability of
very big training data, the expressiveness of big deep models, and the
computational power of GPU clusters. However, they are double-edged
swords: it is costly or sometimes impossible to acquire sufficient labeled
data for training; big models are usually hard to train and might exceed
the capacity of GPU devices; it is non-trivial to distribute the training
onto multiple nodes, with linear speed up and without accuracy loss. In
this talk, I will introduce our recent research to address these
challenges. First, I will introduce a technology called "dual learning",
which leverages the fact that many AI tasks have dual forms to create a
closed feedback loop to enable the effective learning from unlabeled data.
Second, we study the case that deep learning model is large due to its fat
output layer (i.e., with many categories to predict), and propose to map
the outputs onto a 2-dimensional table to effectively compress the model.
By taking recurrent neural networks (RNN) as example, we show that our
technology can lead to better accuracy and several-orders-of-magnitude
smaller model. Third, we discuss the embarrassment of parallel computation
- synchronous parallelization is slow due to synchronization barrier;
asynchronous parallelization hurts accuracy due to communication delay. We
then introduce a novel technology that leverages Taylor expansion of the
gradient function to compensate the delay in asynchronous parallelization.
It can achieve linear speed up and an accuracy comparable to sequential
algorithms. All the technologies introduced in this talk will soon be
open-sourced through Microsoft CNTK.


*******************
Biography:

Tie-Yan Liu is a principal researcher of Microsoft Research Asia, leading
the research on artificial intelligence and machine learning. He is very
well known for his pioneer work on learning to rank and computational
advertising, and his recent research interests include deep learning,
reinforcement learning, and distributed machine learning. As a researcher
in an industrial lab, Tie-Yan is making his unique contributions to the
world. On one hand, many of his technologies have been transferred to
Microsoft's products and online services, such as Bing, Microsoft
Advertising, and Azure. On the other hand, he has been actively
contributing to academic communities. He is an adjunct/honorary professor
at Carnegie Mellon University (CMU), University of Nottingham, and several
other universities in China. His papers have been cited for tens of
thousands of times in refereed conferences and journals. He has won quite
a few awards, including the best student paper award at SIGIR (2008), the
most cited paper award at Journal of Visual Communications and Image
Representation (2004-2006), the research break-through award at Microsoft
Research (2012), and Top-10 Springer Computer Science books by Chinese
authors (2015). He has been invited to serve as general chair, program
committee chair, or area chair for a dozen of top conferences including
SIGIR, WWW, KDD, NIPS, IJCAI, AAAI, ICTIR, as well as associate
editor/editorial board member of ACM Transactions on Information Systems,
ACM Transactions on the Web, Neurocomputing, Information Retrieval
Journal, and Foundations and Trends in Information Retrieval. Tie-Yan Liu
is a fellow of the IEEE, a distinguished scientist of the ACM, a senior
member and academic committee member of the CCF, and a vice chair of the
CIPS information retrieval technical committee.