TOWARDS ASSESSING AND ENHANCING MODERN SOFTWARE REVERSE ENGINEERING

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


PhD Thesis Defence


Title: "TOWARDS ASSESSING AND ENHANCING MODERN SOFTWARE REVERSE ENGINEERING"

By

Mr. Zhibo LIU


Abstract:

Software reverse engineering is the process of converting incomprehensible
binary software into human-readable or analysis-friendly high-level
representation. In many securitycritical tasks, the source code is usually
unavailable, which hinders automatic analysis and makes further fixing or
hardening impossible. In order to be able to analyze software in such
scenarios, the methodologies of software reverse engineering are summarized and
developed to extract the essence of the program while discarding
machine-specific details.

After decades of continuous development and the investment of enormous
resources, perfect reverse engineering is still beyond reach due to the
complexity of binaries. This thesis presents our works of systematically
testing and evaluating modern reverse engineering tools. Our works provide a
deep and timely understanding of these tools, pointing out existing problems
and possible future directions. Based on the observations provided by our
works, we also extend reverse engineering techniques to the emerging field of
DNN executables.

Our first work tests decompilation correctness to present an up-to-date
understanding of modern C decompilers. Our decompiler testing framework
identifies 13 bugs in two open-source decompilers. Moreover, our findings
reveal that modern decompilers are making promising progress in functional
correctness and have been underestimated by researchers. Nevertheless, we show
that some tasks that have been studied for years in academia, such as variable
recovery and type inference, still impede C decompilers from generating quality
outputs.

Our second work conducts an in-depth study of binary lifters from the
"expressiveness" perspective. We demystify binary lifters and reveal how well
the lifters' output can support security-critical downstream applications. We
study four popular static and dynamic LLVM IR lifters that were developed by
the industry or academia from a total of 252,063 executables generated across
compilers, optimizations, and architectures. Our findings show that such binary
lifters are suitable for common similarity- or code comprehension-based
security analysis (e.g., binary diffing). However, the lifted IR code appears
unsuited to rigorous static analysis (e.g., pointer analysis). We summarize our
findings and suggest the correct use and further enhancement of binary lifters.

Our third work presents BTD (Bin to DNN), a decompiler for deep neural network
(DNN) executables. BTD takes DNN executables and recovers DNN model
specifications, including DNN operators, network topology, dimensions, and
parameters. BTD delivers a practical framework to process DNN executables
compiled by different deep learning compilers and with full optimizations
enabled on x86 platforms.

Finally, in our fourth work, we investigate the possibility of reverse
engineering compiled DNN executables in a remote scenario without physical
access to the victim device. All existing methods are limited and can hardly be
extended to reverse model architecture from DNN executables. Nevertheless, with
in-depth analysis, we discover that cacheaware optimizations employed by DL
compilers result in distinguishable DNN operator cache access traces and make
model architecture recovery possible. In this paper, we propose DEEPCACHE, a
novel end-to-end, machine learning-based attack to reverse complex DNN model
architectures deployed on the clouds.


Date:                   Wednesday, 23 August 2023

Time:                   1:00pm - 3:00pm

Venue:                  Room 3494
                        lifts 25/26

Chairperson:            Prof. Richard LAKERVELD (CBE)

Committee Members:      Prof. Shuai WANG (Supervisor)
                        Prof. Lionel PARREAUX
                        Prof. Charles ZHANG
                        Prof. Wei ZHANG (ECE)
                        Prof. Xiapu LUO (PolyU)


**** ALL are Welcome ****