From Language and Vision to Actions: Tool Integration, Planning, and Scaling in Multimodal Agents

PhD Thesis Proposal Defence


Title: "From Language and Vision to Actions: Tool Integration, Planning, and
Scaling in Multimodal Agents"

by

Mr. Zhaoyang LIU


Abstract:

Multimodal agents aim to complete tasks by interpreting language instructions,
perceiving visual inputs, and interacting with their environments. In this
thesis, we investigate effective tool use in multimodal agents through reliable
tool orchestration, unified model--tool communication, and scalable learning for
computer interaction. First, ControlLLM addresses reliable orchestration for
multi-step multimodal tasks. It explicitly models tool--resource dependencies
and combines language-model reasoning with graph search to construct executable
plans, improving the reliability of tool selection, argument assignment, and
execution. Second, VisionLLM v3 unifies tool invocation across three levels:
built-in capabilities, differentiable decoders, and external tools. We
investigate message transmission between the vision-language model and tools
through learned representations and textual arguments. This framework enables
joint optimization and flexible tool invocation through a unified interface.
Third, ScaleCUA studies how multimodal agents learn to interact with dynamic
environments using visual observations and language instructions in computer-use
tasks. A multi-platform interaction data pipeline and unified action space
support GUI understanding, grounding, and task completion, providing a scalable
foundation for agents across desktop, mobile, and web environments. Generally,
these studies establish complementary foundations for multimodal task completion
through dependency-aware planning, unified tool integration, and scalable
interaction learning, advancing multimodal agents from understanding
instructions to acting in complex environments.


Date:                   Tuesday, 3 November 2026

Time:                   2:00pm - 4:00pm

Venue:                  Room 3494
                        Lifts 25/26

Committee Members:      Dr. Qifeng Chen (Supervisor)
                        Dr. Dan Xu (Chairperson)
                        Dr. Yangqiu Song