Efficient KV Cache Management for LLM Serving Systems

PhD Qualifying Examination


Title: "Efficient KV Cache Management for LLM Serving Systems"

by

Miss Qijing LI


Abstract:

Key-value (KV) cache reduces redundant computation in large language model 
(LLM) inference by saving intermediate attention states. Modern LLM serving 
systems retain KV cache of completed requests for future use, but they face 
increasingly long contexts and high user concurrency, which substantially 
enlarge the aggregate KV cache footprint and create pressure on GPU memory 
capacity. To this end, multiple KV cache management systems are proposed to 
efficiently coordinate the storage footprint.

In this survey, we present an up-to-date and thorough introduction to existing 
KV cache management systems. We first provide the background of LLM serving 
and KV caching, including the indexing, allocation, and calculation of KV 
cache entries. Then we review existing hierarchy and distributed KV cache 
storage architectures, which are designed to accommodate large volumes of KV 
cache data. On top of these architectures, we focus on the cache placement 
policies and loading optimizations that aim to retain important KV cache 
entries in faster storage tiers while minimizing retrieval overhead from 
slower tiers. Finally, we discuss attention-specific KV cache management, 
which is tailored to different attentions' KV presentations and reuse 
patterns.


Date:                   Wednesday, 28 October 2026

Time:                   9:00am - 11:00am

Venue:                  Room 5504
                        Lifts 25/26

Committee Members:      Prof. Kai Chen (Supervisor)
                        Dr. Chaojian Li (Chairperson)
                        Dr. Binhang Yuan