Evolving LLM Reinforcement Learning Workloads: Toward RL-Native Training Engines

PhD Qualifying Examination


Title: "Evolving LLM Reinforcement Learning Workloads: Toward RL-Native 
Training Engines"

by

Mr. Pengbo LI


Abstract:

Reinforcement learning (RL) has become a central stage in the post-training 
of large language models (LLMs). Its workloads have shifted rapidly from 
short-horizon alignment with human feedback (RLHF), through reasoning with 
verifiable rewards (RLVR), to long-horizon agents that interact with external 
environments, compact their own context, and coordinate with other agents. 
The prevailing system architecture has changed far less: an inference engine 
generates trajectories, and a training engine inherited from pretraining 
updates the policy. This survey argues that the mismatch between these 
workloads and the assumptions of pretraining-oriented training engines, 
namely static batches of independent sequences executed in synchronous and 
stateless steps, has become a first-order systems problem.

We first characterize the evolution of LLM RL by workload structure rather 
than by algorithm, tracing how trajectory topology, state lifetime, sample 
sharing, and synchronization requirements change from RLHF to multi-agent 
execution. We then identify five challenges for training engines: 
rollout-training consistency, dynamic batching, placement of long-observation 
prefill, training-time prefix reuse, and tree- and DAG-structured 
trajectories. We find that existing solutions are fragmented across 
orchestration layers, kernels, and algorithms; what is missing is an 
execution layer that carries trajectory semantics, policy provenance, and 
reusable state into the trainer. As a case study of such an RL-native 
primitive, we present our system PrefixFlow (ACM SIGOPS ATC 2026). PrefixFlow 
introduces a training-time KV cache that turns shared-prefix reuse into a 
three-phase schedule: one prefix forward pass, suffix microbatches that 
accumulate prefix key/value gradients, and one delayed prefix backward pass. 
It matches full-sequence training within BF16 tolerance, reduces GRPO step 
time by 1.90x (6.1x for the actor update), and lowers suffix-phase peak 
memory by up to 59%. We conclude with open problems for training engines that 
treat trajectory structure, policy versions, and KV state as first-class 
objects.

Keywords: large language models; reinforcement learning; RL training systems; 
agentic RL; prefix reuse; KV cache.


Date:                   Friday, 23 October 2026

Time:                   9:00am - 11:00am

Venue:                  Room 3494
                        Lifts 25/26

Committee Members:      Prof. Kai Chen (Supervisor)
                        Dr. Yangqiu Song (Chairperson)
                        Dr. May Fung