Efficient and Adaptive Systems for Large Language Models

PhD Thesis Proposal Defence


Title: "Efficient and Adaptive Systems for Large Language Models"

by

Mr. Ran YAN


Abstract:

Improving the computational efficiency of large language model (LLM) training 
and inference is critical to reducing resource requirements and enabling 
scalable deployment. Efficiency challenges arise at both the framework level, 
where distributed systems coordinate computation and memory across devices, 
and the kernel level, where operator implementations determine GPU 
utilization. Distributed deployment is central to using large GPU clusters 
effectively, while attention accounts for substantial computation as context 
lengths grow. Heterogeneous training and inference offer a promising approach 
to exploiting diverse GPU resources, while block-sparse attention is a widely 
studied approach to reducing computational cost by limiting token 
interactions.

At the framework level, prior state-of-the-art systems such as Megatron rely 
on largely symmetric parallel configurations, limiting their ability to adapt 
workload allocation to heterogeneous GPU capabilities. To address this 
limitation, we design HexiScale, a heterogeneous LLM training system that 
combines asymmetric data and pipeline parallelism with a hierarchical 
graph-partitioning scheduler to accommodate differences in GPU compute 
capability, memory capacity, and network connectivity. With approximately the 
same total theoretical FLOPS, HexiScale on heterogeneous GPUs achieves an 
average of 83% and up to 101% of the throughput of Megatron and Galvatron on 
homogeneous Ethernet clusters in our 7B-30B model experiments.

At the kernel level, Native Sparse Attention (NSA), a representative 
state-of-the-art block-sparse method, combines compressed representations, 
dynamically selected key-value blocks, and local sliding windows. However, 
its token selection attention kernel relies on large grouped-query attention 
(GQA) groups for efficient execution, limiting its efficiency across LLMs 
with fewer query heads per key-value head. To extend efficient NSA execution 
to a wider range of LLM and GQA configurations, we design Flash Sparse 
Attention (FSA). FSA inverts NSA's kernel loop order, processing key-value 
blocks in the outer loop and batches of query tokens attending to each block 
in the inner loop. This reordering reduces padding and improves matrix 
utilization and key-value reuse while preserving NSA's sparse attention 
pattern. Compared to the evaluated NSA implementation, FSA achieves up to 
3.5x kernel speedup on H20 GPUs. Across our LLM workloads, it achieves up to 
1.25x end-to-end training speedup and 1.36x inference prefill speedup over 
NSA.


Date:                   Monday, 26 October 2026

Time:                   10:00am - 11:00am

Venue:                  Room 3494
                        Lifts 25/26

Committee Members:      Dr. Binhang Yuan (Supervisor)
                        Dr. Chaojian Li (Chairperson)
                        Dr. Wei Wang