A Survey of Diffusion Language Models: From Modeling to Scaling

PhD Qualifying Examination


Title: "A Survey of Diffusion Language Models: From Modeling to Scaling"

by

Mr. Fangyu DING


Abstract:

Autoregressive language models generate text by predicting one token at a time 
from left to right. While this sequential formulation is simple and effective, 
it introduces an intrinsic inference bottleneck: generating a long sequence 
requires thousands of sequential forward passes, and tokens cannot be easily 
revised once emitted. Diffusion language models offer a fundamentally 
different perspective. Instead of generating tokens sequentially, they 
initialize the entire sequence as noise and iteratively refine all positions 
in parallel. This formulation provides natural advantages: parallel decoding, 
full bidirectional context, and, depending on the noise process, iterative 
revision or constraint satisfaction.

This survey provides an accessible synthesis of diffusion language models, 
from their continuous-time mathematical foundations through four modeling 
paradigms to foundation-scale, hardware-compatible serving. We begin with a 
unified continuous-time stochastic framework that covers both continuous 
diffusion and discrete Markov chains. Within this framework, we systematically 
dissect four foundational modeling paradigms:

(1) Mask-Based Diffusion, which corrupts tokens toward an absorbing mask state 
and simplifies score matching into weighted cross-entropy; (2) Insertion-Based 
Diffusion, which models variable-length text via continuous-time deletion and 
insertion; (3) Uniform-State Diffusion, which substitutes tokens toward 
uniform noise; Duo's argmax duality is used for a training curriculum and for 
Discrete Consistency Distillation, while generation remains discrete 
substitution; and (4) Continuous Latent Diffusion, which embeds discrete 
tokens into a low-dimensional codebook and applies continuous Gaussian 
diffusion with PF-ODE sampling. For each paradigm, we derive its training 
objectives, analyze its theoretical properties, and present its core sampling 
mechanics in a clear, step-by-step manner.

Moving from theory to practice, we examine how these formulations scale to 
billions of parameters. We trace the shift from full-sequence bidirectional 
diffusion, which preserves global context but incurs a memory wall, to block 
diffusion, which restores prefix Key-Value caching and enables continued 
pretraining from mature autoregressive checkpoints. We further review how 
masked diffusion can be reparameterized as purely causal attention for 
standard serving engines, together with adaptive block sizing that avoids 
wasted computation on already-resolved tokens.

Finally, we discuss key open challenges, including whether more flexible 
modeling paradigms can surpass mask-based block diffusion, high- concurrency 
serving economics, the factorization gap in parallel decoding, and 
post-training without exact sequence likelihoods. By clarifying the core 
mechanics and practical trade-offs of diffusion language models, this survey 
aims to provide a clear and rigorous foundation for research in 
non-autoregressive sequence generation.


Date:                   Monday, 28 September 2026

Time:                   3:00pm - 5:00pm

Venue:                  Room 3494
                        Lift 25/26

Committee Members:      Dr. Binhang Yuan (Supervisor/Chairperson)
                        Dr. Chaojian Li
                        Dr. Shuai Wang