Interactive Visual Analysis of Human Behavior-Oriented Videos

PhD Thesis Proposal Defence


Title: "Interactive Visual Analysis of Human Behavior-Oriented Videos"

by

Mr. Haipeng ZENG


Abstract:

Analyzing human behaviors in videos has great value for various 
applications, such as education, communication, sports, and surveillance. 
For example, analyzing students' engagement in classroom videos can help 
teachers improve teaching and analyzing speakers' presentation skills in 
presentation videos can facilitate presentation skills training. However, 
it is very time-consuming to manually digest human behavior-oriented 
videos, especially when users need to conduct detailed analysis, such as 
dynamic behavior comparison and behavior evolution exploration. Therefore, 
recent research has proposed automated video analysis techniques to 
facilitate this process, such as face detection, emotion recognition, pose 
estimation and action recognition. Although they have demonstrated 
promising performances in extracting human behaviors, in the real world 
they are insufficient to support detailed analysis with various analytical 
tasks. To this end, visual analytics has been applied to effectively 
analyze huge information spaces, support data exploration and facilitate 
decision-making, which sheds light on helping users interactively explore 
and analyze human behavior-oriented videos.

In this thesis, we propose three novel interactive visual analytics 
systems that combine automated video analysis techniques with 
human-centered visualizations to help users explore and analyze human 
behavior-oriented videos. In our first work, we propose EmotionCues, a 
visual analytics system that integrates emotion recognition algorithms 
with visualizations to easily analyze classroom videos from the 
perspective of emotion summary and detailed analysis. In particular, the 
system supports the visual analysis of classroom videos on two different 
levels of granularity, namely, the overall emotion evolution patterns of 
all the involved people, and the detailed visualization of an individual's 
emotions. In the second work, considering the multi-modality of video 
data, we propose EmoCo, an interactive visual analytics system to 
facilitate the fine-grained analysis of emotion coherence across face, 
text, and audio modalities in presentation videos. By developing suitable 
interactive visualizations enhanced with new features, the system allows 
users to conduct an in-depth exploration of emotions on three levels of 
details (i.e., video, sentence, word level). In the third work, we focus 
on visualizing hand movement in videos and propose GestureLens, a visual 
analytics system to help users explore and analyze gesture usage in 
presentation videos. It enables users to gain a quick spatial and temporal 
overview of gestures, as well as to conduct both content-based and 
gesture-based explorations. Both real-world case studies and feedback from 
the collaboration domain experts verify the effectiveness and usefulness 
of all the proposed systems.


Date:                   Monday, 4 May 2020

Time:                   3:00pm - 5:00pm

Zoom Meeting:           https://hkust.zoom.us/j/94290432651

Committee Members:      Prof. Ting-Chuen Pong (Supervisor)
                        Prof. Huamin Qu (Supervisor)
                        Dr. Xiaojuan Ma (Chairperson)
                        Dr. Pedro Sander


**** ALL are Welcome ****