Efficient Video Object Segmentation with Space-Time Correspondence Networks

MPhil Thesis Defence


Title: "Efficient Video Object Segmentation with Space-Time Correspondence 
Networks"

By

Mr. Ho Kei CHENG


Abstract

We present a simple yet effective approach to modeling space-time 
correspondences in the context of video object segmentation. Unlike most 
existing approaches, we establish correspondences directly between frames 
without re-encoding the mask features for every object, leading to an efficient 
and robust framework. With the correspondences, every node in the current query 
frame is inferred by aggregating features from the past in an associative 
fashion. We cast the aggregation process as a voting problem and find that the 
existing inner-product affinity leads to poor use of memory with a small 
(fixed) subset of memory nodes dominating the votes, regardless of the query. 
With our proposed negative squared Euclidean distance, every memory node now 
has a chance to contribute, and such diversified voting is beneficial to both 
memory efficiency and accuracy.

Next, we present a novel modular paradigm to incorporate user interactions in 
the process by decoupling interaction-to-mask and mask propagation, allowing 
for higher generalizability and better performance. Trained separately, the 
interaction module converts user interactions to an object mask, which is then 
temporally propagated by our propagation module. To effectively take the user’s 
intent into account, a difference-aware fusion module is used to align target 
features with space-time attention.

We also contribute a large-scale, pixel-accurate, and synthetic dataset BL30K 
which can be used for pretraining for a further performance boost. The 
resultant model achieves state-of-the-art results in both semi-supervised mask 
propagation and interaction video object segmentation settings with a fast 
running time.


Date:  			Thursday, 29 July 2021

Time:			2:30pm - 4:30pm

Zoom meeting: 
https://hkust.zoom.us/j/93674871299?pwd=cjE3SkZnUVE2UEVBQzlJR3Jwc2NxZz09

Zoom meeting venue: 	Room 3494
 			Lifts 25/26

Committee Members:	Prof. Chi-Keung Tang (Supervisor)
 			Dr. Qifeng Chen (Chairperson)
 			Dr. Sai-Kit Yeung (ISD)


**** ALL are Welcome ****