Visual Analytics Approaches for Multimodal Temporal Data Analysis

PhD Thesis Proposal Defence


Title: "Visual Analytics Approaches for Multimodal Temporal Data Analysis"

by

Mr. Kam-Kwai WONG


Abstract:

Many real-world analytic tasks focus on time-series data that are enriched by
additional modalities such as text, audio, images, or domain knowledge.
However, integrating these heterogeneous data sources for temporal analysis
poses significant challenges. Differences in data format, scale, granularity,
and semantics make it difficult for analysts to connect events across
modalities and derive meaningful insights over time. This thesis proposal
addresses these challenges through visual analytics approaches that combine
multiple data modalities with temporal data to augment human reasoning about
temporal phenomena. It advances a simple premise: a time series rarely
captures the full complexity of the phenomenon, and its interpretability and
decision value increase when paired with a complementary modality.

I demonstrate the approach through the design and development of three novel
visual analytics systems, each tackling a distinct domain problem. All three
systems were created following a human-centered design study methodology in
collaboration with domain experts. First, we introduce Anchorage, a visual
analytics system for evaluating customer satisfaction in video-based customer
service interactions. Anchorage summarizes multimodal behavioral features from
service videos (e.g., facial expressions) and reveals abnormal service
operations via intuitive visualizations. By structuring videos around key
"anchor events," the system helps service providers quickly navigate long
recordings and assess customer satisfaction at both overall service and finer
operation levels. Second, we present Prismatic, a visual analytics system
that integrates quantitative time-series performance data with qualitative
business knowledge to analyze concept stocks in finance. Prismatic enables
interactive clustering of related stocks through a coordinated multi-view
interface, combining data-driven correlations with knowledge-driven
similarities (e.g., industry relationships) for a cross-validated
understanding of market trends. Third, we develop LandSAR, an immersive
analytics prototype for exploring landslide simulation data in a geospatial
context. LandSAR merges Augmented Reality (AR) visuals, tangible 3D terrain
models, and user interaction to provide an engaging, situated visualization
of temporal landslide dynamics and their environmental impacts. This
multimodal approach provides embodied understanding to general audiences of
how a landslide unfolds over time. This thesis proposal contributes new
visual analytics techniques and systems that integrate multimodal data for
temporal reasoning in diverse application areas. The findings illustrate how
combining heterogeneous data sources, from video and audio to knowledge
representations and physical models, can increase the information density and
interpretability of time-series analysis, leading to deeper insights and more
informed decision-making.


Date:                   Thursday, 16 October 2025

Time:                   10:00am - 12:00noon

Venue:                  Room 4475
                        Lifts 25/26

Committee Members:      Prof. Hongbo Fu (Chairperson)
                        Prof. Huamin Qu (Supervisor)
                        Dr. Arpit Narechania