Turning the Physical World into a Data Interface: Augmenting Situated Visualization and Data Storytelling through Physical Affordances

PhD Thesis Proposal Defence


Title: "Turning the Physical World into a Data Interface: Augmenting Situated
Visualization and Data Storytelling through Physical Affordances"

by

Mr. Kentaro TAKAHIRA


Abstract:

In our daily lives, data is embedded in the physical environments we inhabit. 
However, interacting with this data remains challenging because conventional 
visualization paradigms are primarily designed for two-dimensional screens. 
When applied to the physical world, these screen-based interactions often 
fail to leverage users' natural embodied cognition and struggle to adapt to 
diverse, opportunistic environments. This disconnect results in significant 
cognitive and manipulative friction during everyday sensemaking and 
synchronous data storytelling.

To address this pervasive challenge, this thesis proposes turning the 
physical world itself into an active data interface. By systematically 
mapping and leveraging a spectrum of physical affordances, ranging from 
spatial configurations and tangible manipulations to the semantic states of 
everyday objects, this approach transforms passive environments into dynamic 
media for data interaction. Drawing on Augmented Reality (AR), Computer 
Vision, and Generative AI, this research establishes comprehensive frameworks 
to seamlessly synchronize users' physical actions with data visualizations.

I investigate this physical-digital integration through three interconnected 
systems. The first study introduces VisTranslator, a multimodal AR system 
that addresses everyday situated sensemaking. Driven by a novel Large 
Language Model (LLM) pipeline, it interprets unstructured, physically 
embedded data to dynamically generate in-situ visualizations, allowing users 
to interpret and interact with static embedded data in a personalized, 
context-aware way through object-based manipulations. The second study 
explores tangible interaction for structuring complex data during synchronous 
storytelling. I present TangibleNet, an AR prototype that employs physical 
magnets as tangible proxies for network nodes. This system enables presenters 
to intuitively filter and restructure node-link diagrams in real time, 
creating a direct, low-cognitive-load link between physical manipulation and 
digital control. The third study proposes semantic coupling to achieve 
seamless integration between physical objects and their associated data 
visualizations. InSituTale introduces the novel concept of augmented physical 
data storytelling, leveraging the spatial configurations and semantic states 
of everyday objects to trigger context-aware visualization changes.

Ultimately, this thesis provides empirical evidence and actionable design 
implications demonstrating how utilizing the inherent affordances of physical 
objects can fundamentally enhance intuitive, expressive data interactions 
across varied physical contexts.


Date:                   Monday, 28 April 2026

Time:                   10:00am - 12:00pm

Venue:                  Room 2132C
                        Lift 22

Committee Members:      Prof. Huamin Qu (Supervisor)
                        Dr. Xiaojuan Ma (Chairperson)
                        Dr. Tristan Braud