RLTrace: Synthesizing High-Quality System Call Traces for OS Fuzz Testing

MPhil Thesis Defence


Title: "RLTrace: Synthesizing High-Quality System Call Traces for OS Fuzz 
Testing"

By

Miss Wei CHEN


Abstract

Securing operating system (OS) kernel is one central challenge in today’s cyber 
security landscape. The cutting-edge testing technique of OS kernel is software 
fuzz testing. By mutating the program inputs with random variations for 
iterations, fuzz testing aims to trigger program crashes and hangs caused by 
potential bugs that can be abused by the inputs.

To achieve high OS code coverage, the de facto OS fuzzer typically composes 
system call traces as the input seed to mutate and to interact with OS kernels. 
Hence, quality and diversity of the employed system call traces become the 
prominent factor to decide the effectiveness of OS fuzzing. However, these 
system call traces to date are generated with hand-coded rules, or by analyzing 
system call logs of OS utility programs. Our observation shows that such system 
call traces can only subsume common usage scenarios of OS system calls, and 
likely omit hidden bugs.

In this research, we propose a deep reinforcement learning-based solution, 
called RLTrace, to synthesize diverse and comprehensive system call traces as 
the seed to fuzz OS kernels. During model training, the deep learning model 
interacts with OS kernels  and infers optimal system call traces w.r.t. our 
learning goal — maximizing kernel code coverage. Our evaluation shows that 
RLTrace outperforms other seed generators by producing more comprehensive 
system call traces, subsuming system call corner usage cases and subtle 
dependencies. By feeding the de facto OS fuzzer, Syzkaller, with system call 
traces synthesized by RLTrace, we show that Syzkaller can achieve higher code 
coverage for testing Linux kernels. Furthermore, RLTrace found one 
vulnerability in the Linux kernel (version 5.5-rc6; released on 12 January 
2020), which is publicly unknown to the best of our knowledge.


Date:  			Thursday, 22 July 2021

Time:			3:00pm - 5:00pm

Zoom meeting: 
https://hkust.zoom.us/j/97404189489?pwd=ZXFGeEFVdnh6MUhpcWxSdXRUTU5QQT09

Committee Members:	Dr. Shuai Wang (Supervisor)
 			Dr. Wei Wang (Chairperson)
 			Dr. Amir Goharshady


**** ALL are Welcome ****