Bridging Intent, Execution, and Evaluation in Human-Agent Collaboration: A Two-Sided Survey

PhD Qualifying Examination


Title: "Bridging Intent, Execution, and Evaluation in Human-Agent 
Collaboration: A Two-Sided Survey"

by

Mr. Yanwei HUANG


Abstract:

Large language models have shifted interactive AI systems from single-turn 
answer generation toward agents that interpret user goals, plan multi-step 
procedures, invoke tools, manipulate external artifacts, and revise their 
behavior over extended interactions. This shift introduces a task-level 
alignment challenge: human instructions are often incomplete, ambiguous, or 
evolving, while agents must transform them into concrete execution 
trajectories whose intermediate decisions may be difficult for users to 
observe or correct. We characterize this process as an intent--execution-- 
outcome loop and identify three recurring sources of misalignment: an intent 
gap between what the user wants and what the agent understands, an execution 
gap between the agent's interpretation and the actions it successfully 
carries out, and a control gap between agent behavior and the user's ability 
to inspect, evaluate, and intervene in that behavior.

This report surveys recent approaches for mitigating these gaps in 
human--agent collaboration, with a particular focus on two representative 
downstream domains: software coding and creative generation. We organize the 
literature according to the primary locus and mechanism of intervention. On 
the agent side, we examine learning-based methods that adapt model behavior 
through training, external-knowledge-based methods that augment agents with 
retrieved knowledge, structured resources, or tools, and orchestration-based 
methods that coordinate multiple agents or specialized roles. On the user 
side, we distinguish Conversational Interaction, which uses dialogue to 
elicit, refine, and correct user intent, from Direct Manipulation, which 
exposes persistent representations of intents, artifacts, intermediate 
states, or execution trajectories for inspection and intervention. Across the 
surveyed work, we compare how these mechanisms address different stages of 
the intent--execution--outcome loop and identify several open directions, 
including persistent and revisable intent modeling, reversible and risk-aware 
execution, adaptive allocation of human and agent initiative, and evaluation 
methods that measure the quality of collaboration rather than task success 
alone. The broader goal is to clarify how agent capabilities and interaction 
design can jointly support more reliable, inspectable, and controllable 
LLM-based agents.


Date:                   Monday, 26 October 2026

Time:                   3:00pm - 5:00pm

Venue:                  Room 5504
                        Lifts 25/26

Committee Members:      Dr. Arpit Narechania (Supervisor)
                        Dr. Xiaojuan Ma (Chairperson)
                        Dr. Anyi Rao