Collaborative Sensemaking with AI: From In-Situ Context to Human Experience Understanding

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


PhD Thesis Defence


Title: "Collaborative Sensemaking with AI: From In-Situ Context to
Human Experience Understanding"

By

Mr. Junze LI


Abstract:

Most human-AI collaborative systems for understanding human experience cast 
AI as a post-hoc analyst: experience is first captured, and only afterward 
handed to the system for analysis once the moment has passed. While effective 
for summarizing what was recorded, this paradigm renders the understanding of 
experience retrospective rather than in-situ, verbal rather than multimodal, 
and solitary rather than shared. In each case, the subjective context that 
gives an experience its meaning, when and where it occurred, what was felt, 
what went unsaid, decays in memory, slips beneath what can be put into words, 
or is misread in private interpretation before analysis can even begin. 
Drawing on theories of sensemaking, this thesis argues that post-hoc analysis 
has become an implicit and largely unexamined constraint on how AI supports 
the study of human experience. I propose moving beyond it by repositioning AI 
as a proactive, in-situ scaffold that recovers subjective context as 
experience unfolds: inferring it from multimodal signals, presenting it for 
human verification, and feeding it back to support sensemaking, while leaving 
the construction of meaning to people. This thesis presents three human-AI 
collaborative systems that instantiate and evaluate this repositioning in 
real user research settings. InsightBridge aligns mental models in 
synchronous interviews through real-time information synthesis and shared 
visual communication. SenseFusion reconstructs affective experience in 
retrospective think-aloud by fusing multimodal sensor data into interpretable 
cues of users' inner states. DiaryHelper anchors memory in longitudinal diary 
studies by enriching sparse in-situ logs with inferred contextual cues. 
Through system design, implementation, and empirical evaluation, this thesis 
shows how moving beyond post-hoc analysis enables a richer and more faithful 
understanding of subjective experience, and outlines design principles for AI 
that scaffold rather than supplant human sensemaking, with implications 
reaching beyond Human-Computer Interaction (HCI) to the future of work and 
research.


Date:                   Thursday, 6 August 2026

Time:                   12:00noon - 2:00pm

Venue:                  Room 3494
                        Lifts 25-26

Chairman:               Prof. Lizhong ZHENG (ECE)

Committee Members:      Dr. Xiaojuan MA (Supervisor)
                        Prof. Andrew HORNER
                        Dr. Arpit NARECHANIA
                        Prof. Yunya SONG (EMIA)
                        Dr. Haiyi ZHU (Carnegie Mellon University)