Beyond Turn Taking: A Survey of Full-Duplex Speech Models

PhD Qualifying Examination


Title: "Beyond Turn Taking: A Survey of Full-Duplex Speech Models"

by

Mr. Peng XIE


Abstract:

Full-duplex speech models have developed rapidly from early two-speaker audio 
continuation to instruction following, multimodal, tool-augmented, and 
deployable real-time dialogue systems. Neural audio codecs, causal speech 
encoders, streaming generation, and increasingly large conversational corpora 
now allow an agent to keep listening while speaking and to adapt an 
unfinished response to newly arriving user audio. This progress has produced 
a diverse field of open models, partially released research systems, and 
proprietary services. It has also exposed a classification problem: the same 
"full-duplex" label is applied to externally interrupted cascades, learned 
interaction controllers, time-aligned token models, and generators that 
maintain persistent user and agent streams. Existing layer-based taxonomies 
consequently place many causally and architecturally different systems in one 
broad category and reveal little about when new evidence reaches the 
generator or how conversational control is implemented. To address this 
problem, we propose Duplex-TC (Duplex Temporal-Control Taxonomy), a two-view 
framework that separates temporal and stream topology-turn-gated, 
interruptible, event-gated, time-synchronous, or continuously 
co-generative-from control exposure and integration: external orchestration, 
a learned controller, an explicit control stream, or an implicit joint-stream 
policy. Realization suffixes, system-scope labels, capability tags, and 
evidence tiers prevent heterogeneous continuous systems and behavior-only 
commercial services from collapsing into new residual bins. We use Duplex-TC 
to reconstruct the development of the field from dGSLM, Parrot, Moshi, and 
SALM-Duplex to the rapid 2026 expansion, including DuplexCascade, MoshiRAG, 
TASTE2, Lychee-FD, GPT-Live, and NemotronLabs VoiceChat. The catalog also 
traces the Qwen Omni, Gemini Live, and OpenAI real-time voice lineages so 
that influential streaming and proprietary boundary systems remain visible 
without being equated with persistent joint-stream models. More than fifty 
open, partially open, and proprietary models and systems are analyzed. We 
further synthesize the central data challenges-natural- overlap scarcity, 
clock and channel integrity, self-speech leakage, synthetic interaction bias, 
multilingual and multi-party deficits, privacy, and licensing-and explain why 
evaluation must combine causal mid-turn probes, event-conditioned latency 
distributions, recovery quality, calibrated human judgments, and reproducible 
runtime traces rather than relying only on response quality or interruption 
accuracy.


Date:                   Tuesday, 6 October 2026

Time:                   1:00pm - 3:00pm

Venue:                  Room 3494
                        Lifts 25/26

Committee Members:      Dr. Yangqiu Song (Supervisor)
                        Prof. Kani Chen (Co-supervisor, MATH)
                        Dr. Binhang Yuan (Chairperson)
                        Dr. May Fung