Benchmarking Software Security Analysis Tools

PhD Qualifying Examination


Title: "Benchmarking Software Security Analysis Tools"

by

Mr. Wai Kin WONG


Abstract:

In recent years, software security has gained significant attention around 
the world due to the rise in high-profile cyberattacks. In response, 
organizations are adopting various measure to enhance the security of their 
services and products, including the use of software security analysis tools. 
These tools are designed to automatically identify bugs and security 
vulnerabilities in software systems, aiding developers and security analysts 
in addressing security issues before they can be exploited by attackers. 
Nowadays, these tools are widely integrated into the software development and 
maintenance lifecycle. However, these tools are not perfect. Static 
analyzers, for instance, often face trade-offs between precision and 
scalability, leading to inherent limitations. They may suffer from various 
design and implementation flaws that restrict their ability to accurately 
detect vulnerabilities, thus deviating from their advertised capabilities. 
Additionally, many vendors do not disclose these limitations, and in some 
cases, may not even be aware of them. Therefore, the security community must 
adopt a scientific and systematic approach to benchmark and evaluate these 
tools.

This survey explores the current state of the art in benchmarking software 
security analysis tools. We begin by introducing the fundamental design of 
these tools and the challenges involved in their design and engineering. We 
then discuss the obstacles that hinder large-scale evaluation of security 
analysis tools. Following that, we review existing benchmarking methodologies 
and testing frameworks used to assess the effectiveness of these tools. 
Finally, we suggest potential future research directions that could drive the 
advancement of software security analysis tools. We believe our survey will 
benefit the security analysis tooling community.


Date:                   Thursday, 12 December 2024

Time:                   2:00pm - 4:00pm

Venue:                  Room 3494
                        Lifts 25/26

Committee Members:      Dr. Shuai Wang (Supervisor)
                        Dr. Daoyuan Wu (Chairperson)
                        Dr. Ngok Lam
                        Dr. Victor Wei