Towards Practical Automated Testing

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


PhD Thesis Defence


Title: "Towards Practical Automated Testing"

By

Miss Mi Jung KIM


Abstract

Testing is an important and necessary activity in developing programs to gain 
confidence in their behaviour, but creating test cases is a laborious and 
time-consuming job to developers. To reduce the manual effort on unit testing, 
much research has been performed on automated unit test generation based on 
various underlying techniques such as random testing and search-based testing. 
These automated test generation tools provide developers with a set of failing 
tests that can possibly detect uncovered bugs. However, automatically-generated 
tests are not widely used in practice because their inputs often fail to reveal 
the real bugs. The reasons include that generated test failures produce too 
many false alarms due to invalid test inputs, and generated tests fail to 
exercise diverse program behaviours and just focus on maximizing coverage. 
These observations provide strong motivations to develop enhanced automated 
testing tools that achieve higher bug detection rate.

To improve the practical usefulness of automated test generation tools, in this 
thesis, we first propose a technique called PAF that tackles the false alarm 
problem. The main reason of a false alarm failure is that the test fails not 
because the test input actually reveals the real bug, but because it violates 
an implicit precondition that is never specified in the code or document. PAF 
analyzes data flow of failing tests (i.e., test failures) with respect to their 
likelihood of violating the target method’s implicit precondition. The set of 
failing tests are prioritized so that the failures that are less likely to 
violate implicit preconditions  are ranked higher. As a result, PAF provides 
developers with a ranked list of failures so that they can diagnose those 
bug-revealing failures first.

Second, to address the lack of test input diversity,  we propose a technique 
called DGI that diversifies generated test inputs to achieve higher bug 
detection rate. Even though existing search-based techniques guide the 
generation toward maximizing code coverage, achieving high code coverage is not 
enough to exercise faulty behaviours because covering the faulty statements 
does not guarantee the propagation of the fault. DGI addresses this limitation 
by diversifying test inputs efficiently. DGI generates new test cases by 
mutating existing inputs in different combinations not only in test object 
state level but also test statement level.

Finally, we conduct an extensive empirical  study with DGI on over 500 real 
world projects. In addition to evaluating DGI as a bug finding tool given a 
single version of the program, we also evaluate it in a regression scenario 
that requires two versions of the program and regression oracles. The results 
show that DGI is effective in detecting bugs in both regression and 
non-regression scenarios, and can significantly increase the bug detection 
ability of generated tests.

Overall, our results show that PAF and DGI can produce test failures with few 
false alarms and achieve high detection rate. This output supports the 
usefulness of our tool in thorough practical settings. Consequently, the 
techniques and toolsets presented in this thesis brings automated test 
generation tools closer to practical usage.


Date:			Friday, 2 August 2019

Time:			11:00am - 1:00pm

Venue:			Room 3494
 			Lifts 25/26

Chairman:		Prof. Philip Mok (ECE)

Committee Members:	Prof. Sung-Hun Kim (Supervisor)
 			Prof. Shing-Chi Cheung (Supervisor)
 			Prof. Andrew Horner
 			Prof. Frederick Lochovsky
 			Prof. Chi-Ying Tsui (ISD)
 			Prof. Doo-Hwan Bae (KAIST)


**** ALL are Welcome ****