Human Agency in the Age of AI: Towards Human-AI Co-Creation with High Control and High Automation

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


PhD Thesis Defence


Title: "Human Agency in the Age of AI: Towards Human-AI Co-Creation with High
Control and High Automation"

By

Mr. Leixian SHEN


Abstract:

Before the advent of Generative AI (GenAI), digital content creation largely 
depended on manual Authoring tools (e.g., Adobe Illustrator), where humans 
acted as operators who exercised precise control through labor-intensive 
operations. Today, multimodal foundation models can generate images, videos, 
animations, and other media from natural language prompts, dramatically 
lowering the barrier to creation. Yet this shift introduces a central 
tension: as AI becomes more powerful and autonomous, human agency—the 
meaningful ownership and control over the creative process—tends to diminish. 
Users risk being downgraded from creative authors to passive spectators.

To address this issue, this thesis asks: How can we scale AI automation 
without sacrificing human agency? I argue that high automation and high 
control are not inherently contradictory goals. Rather, they can be jointly 
achieved when interaction systems provide shared, manipulable representations 
through which human intent can be expressed, inspected, revised, and carried 
forward. This thesis develops this argument through a progression from 
concrete system explorations to theoretical abstraction, expanding the scope 
and temporality of agency across the human-AI co-creation lifecycle.

The research unfolds across four progressive works. First, Data Player 
explores the Automation stage by automatically generating data videos with 
narration-animation interplay from static visualizations and descriptive 
text. In this stage, the human becomes a consumer of AI-generated media: the 
system achieves high efficiency, but the user's agency is compressed into 
providing initial inputs and accepting or rejecting final outputs. This 
exposes the limitation of full automation without fine-grained human input. 
Second, Data Playwright advances to the Alignment stage by introducing 
annotated narration, a paradigm that allows users to embed natural language 
authoring commands directly within narrative text. Here, the human role 
shifts from passive consumer to active director: users can specify what 
should happen, when it should happen, and which visual elements should be 
affected, while the AI handles interpretation, synchronization, and 
rendering. Third, Live Artifacts extends agency into the Adaptation stage by 
introducing persistent generative media whose specifications remain embedded 
in visual layers and can be re-evaluated across time, context, and 
modalities. In this stage, the human becomes an architect of living media: 
rather than authoring a single static output, the creator defines what 
remains stable, what changes, and how generative behaviors propagate across 
the artifact's lifecycle. Finally, the Interaction-Augmented Instruction 
model provides a theoretical Abstraction of these systems by formalizing how 
natural language prompts and graphical interactions jointly construct 
executable instructions for GenAI. At this meta-design level, the human 
becomes an interaction designer: a designer of the representational 
structures, interaction flows, and instruction paradigms through which future 
users can communicate intent to AI systems with both flexibility and 
precision.

Together, these contributions show that human agency in the age of AI is not 
simply preserved by keeping humans in the loop. It must be deliberately 
designed into the representations, interaction structures, and media 
infrastructures through which humans and AI create together. Ultimately, this 
thesis demonstrates that human agency is not what we save from machines; it 
is what we systematically evolve and expand alongside them.


Date:                   Thursday, 25 June 2026

Time:                   3:00pm - 5:00pm

Venue:                  Room 3494
                        Lifts 25/26

Chairman:               

Committee Members:      Prof. Huamin QU (Supervisor)
                        Prof. Pedro SANDER
                        Dr. Arpit NARECHANIA
                        Prof. Celine Yunya SONG (EMIA)
                        Prof. Yuanchun SHI (Tsinghua University)