Memory Isolation in Modern Computing System

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


PhD Thesis Defence


Title: "Memory Isolation in Modern Computing System"

By

Mr. Hongyi LU


Abstract:

Memory isolation serves as a cornerstone of modern computing security, 
protecting systems from unauthorized memory access across different 
components. Current memory isolation approaches encounter challenges in terms 
of implementing fine-grained, high-performance protection. The adoption of 
heterogeneous accelerators further complicates this issue, as it introduces 
divergent programming models while lacking mature isolation mechanisms. This 
thesis aims to address these emerging challenges by developing innovative 
memory isolation mechanisms and investigating the security risks of the new 
heterogeneous processors.

Our first work explores building efficient yet flexible isolation schemes in 
kernel extensions, specifically the Berkeley Packet Filter (BPF). The 
existing BPF verifier has limited completeness and therefore is often 
bypassed, leading to kernel exploits. We propose MOAT, a hardware-based 
isolation scheme that robustly isolates BPF programs within the Linux kernel 
using Intel Memory Protection Keys (MPK). MOAT introduces a novel two-layer 
isolation design to solve the problem of limited hardware keys, enabling 
secure BPF execution with a throughput loss as low as 3%.

Our second work investigates vulnerabilities within GPU TEEs. We uncover MOLE 
attack, which compromises the security of shim-style GPU TEEs by exploiting 
an under-documented GPU-embedded Microcontroller Unit (MCU) in Arm Mali GPUs. 
MOLE demonstrates that the MCU, whose firmware is loaded by the untrusted OS 
can easily bypass GPU TEEs. MOLE highlights the necessity of incorporating 
all internal hardware components into a comprehensive security model when 
designing memory isolation schemes in a heterogeneous system.

Our third work dives into the memory safety in GPU computing. We present 
CuSafe, a practical GPU memory sanitizer designed to detect memory corruption 
in CUDA applications running on NVIDIA GPUs. CuSafe employs a hybrid metadata 
scheme combining pointer tagging with in-band buffer bounds, which can be 
deployed on commodity GPUs. CuSafe achieves efficient and accurate memory 
access validation with an average performance overhead of only 13% and 
negligible memory overhead.


Date:                   Thursday, 20 August 2026

Time:                   2:00pm - 4:00pm

Venue:                  Room 3494
                        Lifts 25/26

Chairman:               

Committee Members:      Dr. Shuai WANG (Supervisor)
                        Dr. Fengwei ZHANG (SUSTech, Co-supervisor)
                        Dr. Wei WANG
                        Dr. Chaojian LI
                        Dr. Chao TANG (ACCT)
                        Prof. Chao SHEN (Xi'an Jiaotong University)