Controllable Video Generation and World Simulation

PhD Thesis Proposal Defence


Title: "Controllable Video Generation and World Simulation"

by

Mr. Yue MA


Abstract:

Controllable video generation is essential for bridging the gap between 
generative AI and practical creative applications. This thesis presents three 
complementary approaches that address distinct dimensions of controllability 
in video synthesis: motion transfer, 4D video creation, and real-time video 
relighting. First, we propose Follow-Your-Motion, a three-stage video motion 
transfer framework that efficiently finetunes video diffusion transformers to 
synthesize complex motion. By decoupling the attention heads into spatial and 
temporal groups, combined with sparse motion sampling and adaptive rotary 
position embeddings, our method achieves state-of-the-art motion fidelity and 
text alignment while being significantly faster than prior approaches. We 
also introduce MotionBench, a comprehensive benchmark for evaluating video 
motion transfer. Second, we present Follow-Your-Creation, a 4D video creation 
framework that reformulates the generation of multi-view consistent video 
content as a video inpainting task. Given a monocular input video, our method 
leverages depth-based point cloud rendering and composite mask training to 
enable both camera retargeting and prompt-based content editing, with a 
self-iterative tuning strategy for handling large camera motions and 
temporal-packing inference for multi-view consistency. Third, we introduce 
LiveLight, the first diffusion-based framework for real-time streaming video 
relighting with interactive 3D lighting control. LiveLight combines a 
lightweight Multi-Plane Light Irradiance adapter for depth-aware illumination 
injection, a geometry-guided feedback branch for structure-preserving 
shading, and a progressive rolling-window streaming strategy for temporally 
coherent, arbitrarily long video relighting. Extensive experiments across all 
three works demonstrate state-of-the-art performance on their respective 
tasks, collectively advancing the frontier of controllable video generation 
and world simulation.


Date:                   Monday, 5 October 2026

Time:                   4:30pm - 6:30pm

Venue:                  Room 3494
                        Lifts 25/26

Committee Members:      Dr. Qifeng Chen (Supervisor)
                        Dr. May Fung (Chairperson)
                        Dr. Dan Xu