Accommodating Long-Context LLM Training over Heterogeneous Environment

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering

Final Year Thesis Oral Defense

Title: "Accommodating Long-Context LLM Training over Heterogeneous 
Environment"

by

LIANG Yan

Abstract:

Training large language models (LLMs) with ultra-long contexts introduces 
critical memory and communication bottlenecks. While existing parallelization 
strategies scale efficiently on homogeneous clusters, their performance 
degrades under hardware heterogeneity. This thesis presents an efficient 
long-context LLM training system designed for mixed-GPU environments. Our 
approach enables non-uniform workload distribution while preserving causal 
attention correctness. Evaluations demonstrate improved throughput on 
heterogeneous clusters compared to existing systems.

Date            : 27 April 2026 (Monday)

Time            : 16:00 - 16:40

Venue           : Room 2132C (near Lift 19), HKUST

Advisor         : Dr. YUAN Binhang

2nd Reader      : Dr. WANG Wei