Towards Applicable Image/Video Matting

PhD Thesis Proposal Defence


Title: "Towards Applicable Image/Video Matting"

by

Miss Yanan SUN


Abstract:

As a primary image/video editing technique, matting plays an important role in 
image/video beautification or secondary creation, film production, game 
development, e-commerce promotion, etc. Traditional matting is limited by 
accuracy, running speed, and applicable scenarios, so it is difficult to be 
widely promoted. Although the development and application of deep learning have 
solved these problems to a large extent, the vigorous upgrading of the 
multimedia industry in recent years has posed more challenges to the 
applicability of these deep learning-based matting methods. Specifically, the 
diversification of media content and application scenarios, the transformation 
of computing power from servers to mobile devices, and the popularity of 
high-definition(HD) / ultrahigh-definition(UHD) display screens are all the 
difficulties to be solved when improving the applicability of matting 
algorithms. This thesis aims to improve the applicability of matting algorithms 
from multiple aspects based on deep learning. It consists of the following 
three studies.

The first study targets on the applicability required by diverse media 
contents. In addition to human, the media content also includes animals and 
plants, daily necessities, buildings, etc. These are able to be the target 
objects in image and video editing. Existing matting techniques either only 
target specific objects, or use the same module to process all objects 
indiscriminately, which leads to the inability of these algorithms to maintain 
their applicability when dealing with diverse media content, thus failing to 
give correct prediction. For this problem, this study proposes Semantic Image 
Matting (SIM) framework. In this method, we introduce the semantic information 
carried by different alpha patterns of objects into the matting algorithm, so 
that it can perform differentiated processing for different contents. The 
evaluation shows that SIM constantly outperforms the baseline on processing 
various objects.

The second study concentrates on the applicability issues introduced by 
increasingly complex user requirements and application scenarios. Existing 
algorithms usually assume that the target of matting is a whole region, 
however, when the target the user wants to process is no longer the whole 
region, but an individual instance, these algorithms are no longer applicable. 
Even the combinations of existing techniques are difficult to give satisfactory 
results. To address this challenge, we propose instance matting (IM). Different 
from previous matting methods that only disentangles the foreground and 
background color, instance matting can decompose the RGB value of each pixel 
into different instance layers, so as to realize the individual editing of each 
instance, which is undoubtedly an important step for image and video editing.

The third study focuses on the applicability of computational power and advent 
HD / UHD display devices. On the one hand, popular mobile devices usually 
cannot provide sufficient computational power; on the other hand, the promotion 
of HD/UHD displays have greatly increased the amount of computations for image 
and video processing. In such a case of limited computational resources, it is 
not feasible for existing matting algorithms to efficiently process 
ultrahigh-resolution images and videos in one shot. In order to solve the 
applicability of the matting algorithm for portable devices, we propose 
SparseMat, which can skip a large number of unnecessary calculations and only 
process important pixels within transitional regions based on sparse 
convolution. Compared with existing algorithms, experiments show that our 
algorithm can save more than 90% of the computation.


Date:			Tuesday, 28 March 2023

Time:                  	10:30am - 12:30pm

Venue:			Room 2128B
  			lift 19

Committee Members:	Prof. Chi-Keung Tang (Supervisor)
 			Dr. Qifeng Chen (Chairperson)
 			Prof. Yu-Wing Tai
 			Dr. Dan Xu


**** ALL are Welcome ****