Scalable Data Systems for Foundation-Model Data Preparation: Programming, Execution, and Large-Scale Data Curation

PhD Qualifying Examination

Title: "Scalable Data Systems for Foundation-Model Data Preparation:
Programming, Execution, and Large-Scale Data Curation"

by

Mr. Xiaochen MA


Abstract:

Foundation models, and large language models in particular, are increasingly 
improved through repeated data rounds: an evaluation exposes a capability gap, 
and models, verifiers, and humans together generate, judge, filter, and mix 
the data that trains the next model. As models take over more of this work, 
the loop becomes recursive: the model helps produce its own next training 
signal, and the data pipeline becomes the unit of iteration. At that point the 
speed and reliability of improvement depend less on any single data algorithm 
than on the data system that runs the loop. This survey argues that such a 
system must make pipelines easy to program, so that a new algorithm arrives as 
an edit to typed, versioned operators; easy to distribute, so that one recipe 
runs across machines, engines, teams, and agents; and easy to optimize, so 
that the system rather than the author chooses an efficient physical plan, all 
on a foundation of reproducibility that makes every round replayable and its 
effect attributable. We review how foundation-model data is prepared, from 
pretraining corpora to post-training and reinforcement-learning rollouts, and 
show that successive waves of data-centric algorithms recombine a small set of 
stable curation primitives: generation, judgment, verification, deduplication, 
selection, mixing, and decontamination. We then compare data flow engines, 
curation toolkits, LLM-powered semantic operators, and rollout systems by how 
they program and execute these primitives. Six gaps recur: quality semantics 
are not runtime objects, global state is rebuilt rather than maintained, 
lineage breaks under dynamic expansion, recipes are tied to one runtime, 
physical plans are chosen by hand, and systems efficiency is reported 
separately from capability gain. We discuss the risks specific to recursive 
loops, such as model collapse, judge bias, and contamination, and close with a 
research agenda and a vision of a compiler-style substrate: one effect- 
annotated logical plan, checked before it runs, lowered by shape to several 
engines, optimized from the profiles of earlier rounds, and replayable by 
construction.


Date:                   Tuesday, 27 October 2026

Time:                   4:00pm - 5:00pm

Venue:                  Room 3494
                        Lifts 25/26

Committee Members:      Dr. Binhang Yuan (Supervisor)
                        Dr. Yangqiu Song (Chairperson)
                        Dr. Wei Wang