Building Robot Foundation Models with Scalable Video-Action Pretraining

Speaker: Dr. Yinghao XU
Assistant Professor
Department of Computer Science and Engineering
The Hong Kong University of Science and Technology (HKUST)

Title: Building Robot Foundation Models with Scalable Video-Action Pretraining

Date: Monday, 12 October 2026

Time: 4:00pm - 5:00pm

Venue: Lecture Theater F (Leung Yat Sing Lecture Theater), near lift 25/26, HKUST

Abstract:

Most robot foundation models inherit their knowledge from vision-language pretraining, yet robot control is fundamentally about how the physical world evolves under actions. This talk argues that video-action pretraining offers a complementary and scalable foundation for robot learning: by learning to predict scene dynamics, a model acquires manipulation priors from large-scale robot and human videos and translates them into executable actions We systematically study how to design effective proxy tasks for scalable pretraining along three dimensions: the latent space, the autoregressive pretraining objective, and in-context learning. We further show that, with asynchronous inference and efficient deployment, the pretrained model enables real-time closed-loop control on real robots. Together, these results suggest that video-action pretraining provides a scalable path toward general-purpose robot foundation models.

Biography:

Yinghao Xu is an Assistant Professor in the Department of Computer Science and Engineering at the Hong Kong University of Science and Technology (HKUST). Previously, he was a Staff Research Scientist at RobbyAnt, working on world models and embodied AI. Before that, he was a postdoctoral researcher at Stanford University His research lies at the intersection of 3D computer vision, generative AI, and embodied AI, with a recent focus on building world models that unify 3D reconstruction, world simulation, and embodied action. His work has been selected as a Best Paper Award candidate at ECCV 2026 and CVPR 2021. He was the recipient of the Yunfan Award at WAIC 2024 and was nominated for the Snap Fellowship in 2022.

Homepage: https://justimyhxu.github.io/