Memory Isolation in Modern Computing System

PhD Thesis Proposal Defence


Title: "Memory Isolation in Modern Computing System"

by

Mr. Hongyi LU


Abstract:

Memory isolation serves as a cornerstone of modern computing security,
protecting systems from unauthorized memory access across different
components. Current memory isolation approaches encounter challenges in
terms of implementing fine-grained, high-performance protection. The
adoption of heterogeneous accelerators further complicates this issue, as it
introduces divergent programming models while lacking mature isolation
mechanisms. This thesis aims to address these emerging challenges by
developing innovative memory isolation mechanisms and investigating the
security risks of the new heterogeneous processors.

Our first work explores building efficient yet flexible isolation schemes in
kernel extensions, specifically the Berkeley Packet Filter (BPF). The
existing BPF verifier has limited completeness and therefore is often
bypassed, leading to kernel exploits. We propose MOAT, a hardware-based
isolation scheme that robustly isolates BPF programs within the Linux kernel
using Intel Memory Protection Keys (MPK). MOAT introduces a novel two-layer
isolation design to solve the problem of limited hardware keys, enabling
secure BPF execution with a throughput loss as low as 3%.

Our second work investigates vulnerabilities within GPU TEEs. We uncover
MOLE attack, which compromises the security of shim-style GPU TEEs by
exploiting an under-documented GPU-embedded Microcontroller Unit (MCU) in
Arm Mali GPUs. MOLE demonstrates that the MCU, whose firmware is loaded by
the untrusted OS can easily bypass GPU TEEs. MOLE highlights the necessity
of incorporating all internal hardware components into a comprehensive
security model when designing memory isolation schemes in a heterogeneous
system.

Our third work dives into the memory safety in GPU computing. We present
CuSafe, a practical GPU memory sanitizer designed to detect memory corruption
in CUDA applications running on NVIDIA GPUs. CuSafe employs a hybrid
metadata scheme combining pointer tagging with in-band buffer bounds, which
can be deployed on commodity GPUs. CuSafe achieves efficient and accurate
memory access validation with an average performance overhead of only 13%
and negligible memory overhead.


Date:                   Monday, 26 January 2026

Time:                   2:00pm - 4:00pm

Zoom Meeting:
https://hkust.zoom.us/j/91960382296?pwd=fo8mBib7TS3KuPMDxDfEuFKUq65Gaa.1

Committee Members:      Dr. Shuai Wang (Supervisor)
                        Dr. Binhang Yuan (Chairperson)
                        Dr. Chaojian Li