Understanding and Supporting Human Feedback Elicitation for Human-Robot Interaction

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


PhD Thesis Defence


Title: "Understanding and Supporting Human Feedback Elicitation for
Human-Robot Interaction"

By

Miss Hanfang LYU


Abstract:

Robots are increasingly deployed in everyday human-centered spaces, where 
aligning their behavior with human values and situational needs relies on 
high-quality human feedback, i.e., human signals that inform or correct robot 
behavior, including preferences, social cues, and corrective actions. 
Eliciting such feedback efficiently, however, remains non-trivial in both 
directions of human-robot communication. On the understanding human side, 
existing work tends to treat human feedback as homogeneous, well-formed 
input, and does not yet offer a fine-grained account of how its form and 
modality shift across the heterogeneous interaction contexts in which it 
arises. On the robot design side, current systems and robot behaviors do not 
sufficiently support users' actual feedback behavior, since non-expert users 
often do not know how to translate their preferences or natural expressions 
into robot-readable signals, while their motivation is further bounded by 
tedious labeling tasks, occupied cognitive resources during competing 
activities, and the limited sympathy that machine-like robots evoke when they 
encounter errors.

This thesis addresses both gaps across three contexts of post-hoc trajectory 
labeling, in-situ social multitasking, and reactive error moments, where each 
work first deepens the understanding of how humans express feedback in that 
context, and then translates the findings into supportive system or behavior 
design. For the post-hoc context, a formative study (n = 12) on 
preference-based reward learning characterizes three labeler challenges in 
criteria formation, detail noticing, and focus maintenance. The resulting 
FARPLS system, equipped with feature-clustered prompting, keyframe markers, 
and real-time attention monitoring, significantly improves labeling 
consistency and engagement without raising cognitive load (n = 42). For the 
in-situ context, an elicitation study (n = 24) in an AR-simulated coffee-chat 
scenario investigates how participants signal 13 intentions and feedback acts 
to four service robots of distinct morphologies (anthropomorphic, zoomorphic, 
grounded technical, and aerial technical) while acting as a speaker or a 
listener. Coding of 624 social cues with retrospective think-aloud interviews 
reveals that conversational role shapes verbal and gestural cue choice 
whereas morphology shapes modality choice via perceived sensing, yielding 
design implications for low-disruption, role- and morphology-aware 
service-robot response. For the reactive context, co-design workshops (n = 
21) and member-checking interviews (n = 18) produce human- and 
animal-inspired expressive error reactions for machine-like robots, surfacing 
a gap between designer intent and observer perception together with the 
conditions that evoke sympathy and helping intention. We implemented human-, 
animal-, and machine-like designs on an interactive WoZ-controlled MR 
platform that embeds them within a continuous robot task workflow, enabling 
real-time operator control while participants interact with shared household 
objects.

In summary, this thesis contributes to human-robot interaction by delineating 
an understanding of how humans provide feedback across the post-hoc, in-situ, 
and reactive interaction contexts, by developing novel interfaces, behavior 
design strategies and system prototypes that support feedback elicitation, 
and by offering empirical insights from formative, elicitation, and co-design 
studies, that inform the robot designs to support human feedback elicitation.


Date:                   Friday, 31 July 2026

Time:                   10:00am - 12:00noon

Venue:                  Room 3494
                        Lifts 25-26

Chairman:               Dr. Jiying LI (OCES)

Committee Members:      Dr. Xiaojuan MA (Supervisor)
                        Prof. Fugee TSUNG (Co-supervisor, IEDA)
                        Prof. Andrew HORNER
                        Dr. Arpit NARECHANIA
                        Dr. Yantao YU (CIVL)
                        Dr. Kening ZHU (CityU)