PromptPoseFX: Authoring Dynamic Pose-Driven Visual Effects via Language and Multimodal Interaction

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


MPhil Thesis Defence


Title: "PromptPoseFX: Authoring Dynamic Pose-Driven Visual Effects via Language
and Multimodal Interaction"

By

Miss Kexin ZHENG


Abstract:

Authoring pose-driven visual effects with traditional video editing tools is a 
complex and time-consuming task that requires considerable technical expertise 
and manual effort. While generative AI models can simplify the workflow, 
existing approaches primarily focus on pixel-based video frame generation, 
which often alters the original video content and lacks the granularity 
required for fine-grained, iterative refinement.

To address these issues, we present PromptPoseFX, an interactive system that 
allows creators to author pose-driven visual effects through natural language 
and multimodal interaction. Our system leverages Large Language Models (LLMs) 
to translate high- level user intent into parameterized procedural graphics 
scripts, automatically mapping pose-tracking data to specific effect 
parameters. Our script-based approach manages effects as independent layers, 
which preserves the original content of the underlying video. To mitigate the 
ambiguity of natural language, we design a multimodal interface that supports 
(1) direct spatial coordinate selection, (2) temporally grounded prompting, 
and (3) dynamically generated parameter controls. We evaluated our system 
through a comparison study against a conversational interface baseline and an 
exploratory usability study. The results demonstrate that PromptPoseFX offers 
significantly higher controllability and precision than conventional 
conversational interfaces while maintaining a high level of usability, 
effectively lowering the technical barrier to authoring pose-driven visual 
effects.


Date:                   Monday, 2 March 2026

Time:                   2:00pm - 4:00pm

Venue:                  Room 2612A
                        Lifts 31/32

Chairman:               Prof. Pedro SANDER

Committee Members:      Prof. Chiew-Lan TAI (Supervisor)
                        Dr. Xiaojuan MA