Scaling Reasoning to Multimodal Long-Context Language Models

PhD Thesis Proposal Defence


Title: "Scaling Reasoning to Multimodal Long-Context Language Models"

by

Mr. Zhaowei WANG


Abstract:

Reasoning is a key component of artificial general intelligence, with recent
breakthroughs in mathematical problem-solving and program synthesis highlighting
advances in large language models (LLMs). Yet applying these capabilities to
real-world environments goes beyond solving those well-defined problems: models
must make sense of real-world situations, abstract generalizable knowledge from
them, and use this understanding to reason.

Meanwhile, applying this reasoning ability to real-world environments requires
models to understand the visual information they observe. As observations and
interactions accumulate, relevant evidence becomes distributed across
increasingly long multimodal contexts, making its retrieval and use in
subsequent reasoning an additional challenge.

To help LLMs derive generalizable knowledge from real-world situations and use
it for reasoning, we strengthen LLMs' commonsense knowledge through abstraction
and causal reasoning. AbsPyramid evaluates knowledge abstraction through a
unified entailment graph of 221K instances, while AbsInstruct improves LLMs
through explanation-based fine-tuning. Beyond abstraction, COLA mitigates
confounding effects to help LLMs understand causal relations in real-world
situations.

Beyond textual understanding, we equip vision-language models (VLMs) to navigate
embodied environments using visual observations. To this end, we construct
DivScene, a large-scale open-vocabulary object navigation dataset, and fine-tune
navigation agents. The resulting agent outperforms GPT-4o by about 20 percentage
points in success rate. However, its context window holds only recent
observations, restricting access to earlier history.

We therefore turn to developing long-context VLMs. As a first step, we developed
MMLongBench to evaluate diverse long-context vision-language tasks. Evaluation
of 46 models reveals substantial headroom and an association between stronger
reasoning and better long-context performance. Guided by these findings, we will
investigate training data and methods for improving evidence retrieval and
reasoning across long multimodal contexts.

Building on these studies, this proposal aims to extend language model
capabilities from textual understanding to effective reasoning in real-world
environments.


Date:                   Monday, 2 November 2026

Time:                   4:00pm - 6:00pm

Venue:                  Room 4472
                        Lifts 25/26

Committee Members:      Dr. Yangqiu Song (Supervisor)
                        Prof. Raymond Wong (Chairperson)
                        Dr. Mingxun Zhou