Turning the Physical World into a Data Interface: Augmenting Situated Visualization and Data Storytelling through Physical Affordances

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


PhD Thesis Defence


Title: "Turning the Physical World into a Data Interface: Augmenting Situated 
Visualization and Data Storytelling through Physical Affordances"

By

Mr. Kentaro TAKAHIRA


Abstract:

In our daily lives, data is embedded in the physical environments we inhabit.
However, interacting with this data remains challenging because conventional
visualization paradigms are primarily designed for two-dimensional screens.
When applied to the physical world, these screen-based interactions often fail
to leverage users' natural embodied cognition and struggle to adapt to diverse,
opportunistic environments. This disconnect results in significant cognitive
and manipulative friction during everyday sensemaking and synchronous data
storytelling. To address this pervasive challenge, this thesis proposes
turning the physical world itself into an active data interface. By
systematically mapping and leveraging a spectrum of physical affordances,
ranging from spatial configurations and tangible manipulations to the semantic
states of everyday objects, this approach transforms passive environments into
dynamic media for data interaction. Drawing on Augmented Reality (AR) and
Generative AI, the research establishes comprehensive frameworks to
synchronize users' physical actions with data visualizations. The thesis
investigates this physical-digital integration through three interconnected
systems: VisTranslator, a multimodal AR system that interprets unstructured,
physically embedded data to dynamically generate in-situ visualizations; 
TangibleNet, an AR prototype employing physical magnets as tangible proxies
for network nodes to support low-cognitive-load, real-time restructuring of
node-link diagrams during synchronous storytelling; and InSituTale, which
introduces semantic coupling to trigger context-aware visualization changes
based on spatial configurations and object states. Collectively, these
contributions provide empirical evidence and actionable design implications
showing how leveraging the inherent affordances of physical objects can
fundamentally enhance intuitive, expressive data interactions across varied
physical contexts.


Date:                   Tuesday, 14 July 2026

Time:                   10:00am - 12:00noon

Venue:                  Room 3494
                        Lifts 25/26

Chairman:               Prof. Jidong ZHAO (CIVL)

Committee Members:      Prof. Huamin QU (Supervisor)
                        Dr. Tristan BRAUD
                        Dr. Xiaojuan MA
                        Dr. Qiushi ZHOU (CMA)
                        Prof. Takayuki ITOH (Ochanomizu University)