Unified Multimodal Understanding and Generation: Architecture, Representation, and Evaluation

PhD Qualifying Examination


Title: "Unified Multimodal Understanding and Generation: Architecture,
Representation, and Evaluation"

by

Mr. Hongxiang LI


Abstract:

Multimodal understanding and visual generation have long developed along
separate paths, with distinct modeling paradigms and visual representations.
Unified multimodal models aim to integrate both capabilities within a single
model, with the expectation that they reinforce each other. However, the two
tasks place different and often competing demands on a shared model:
understanding relies on abstract semantics, whereas generation requires faithful
pixel-level detail. This raises three fundamental questions: how much of the
model should be shared between the two tasks, which visual representation can
serve both, and how to verify that a model is truly unified rather than two
capabilities placed side by side. In this paper, we systematically review
unified multimodal understanding and generation around these questions, from the
perspectives of architecture, representation, and evaluation. For architecture,
we categorize existing models by the degree of coupling between understanding
and generation, from language models connected to external generators to fully
shared backbones, and analyze the trade-off between knowledge sharing and task
interference. For representation, we examine how semantic and pixel information
are reconciled in visual tokenizers, in the latent spaces of generators, and at
the interface between the language model and the generator. For evaluation, we
review benchmarks for understanding, generation, and their integration, focusing
on whether the knowledge and reasoning a model exhibits in understanding are
consistently reflected in its generation. Finally, we discuss open challenges
and future directions toward genuinely unified multimodal models.


Date:                   Monday, 2 November 2026

Time:                   2:30pm - 4:30pm

Venue:                  Room 5501
                        Lifts 25/26

Committee Members:      Dr. Long Chen (Supervisor)
                        Dr. Dan Xu (Chairperson)
                        Dr. Qifeng Chen