Towards Emulating Interactions for Effective Software Testing

PhD Thesis Proposal Defence


Title: "Towards Emulating Interactions for Effective Software Testing"

by

Mr. Hengcheng ZHU


Abstract:

With the rapid advancement of software technologies, the complexity of 
software systems continues to grow. Modern software systems frequently 
interact with numerous components or external systems, making the effective 
emulation of such sophisticated interactions a fundamental challenge in 
software testing. This thesis tackles this challenge by investigating the 
emulation of interactions at two key testing levels: unit testing and system 
testing.

At the unit testing level, interaction emulation is achieved through test 
doubles, which isolate the component under test from its dependencies. The 
most common role is the stub, which simulates the behavior of substituted 
classes. Developers use stubs to simulate interactions that are complicated 
to set up or difficult to reproduce in automated testing. The emulated 
behavior is encoded in stub code, typically as a sequence of API calls to a 
mocking framework. However, developing and maintaining stub code is 
labor-intensive and error-prone. Therefore, we present StubCoder, a technique 
that automatically generates and repairs stub code for regression testing. 
StubCoder uses an evolutionary algorithm guided by the runtime behavior of 
test cases. Our evaluation shows that StubCoder effectively generates stub 
code for incomplete test cases and repairs obsolete test cases.

In addition to stubs, mocks are another widely used form of test double, 
configured with expectations about their interactions with the SUT. Mocks 
record method calls and enable developers to verify anticipated interactions 
through mock assertions. Mock assertions validate program behaviors that are 
unobservable to test assertions, yet are rarely considered by automated test 
generation techniques. Effective generation of mock assertions requires 
understanding their practical usage. To bridge this gap, we conducted the 
first empirical study of mock assertions, examining their adoption patterns, 
characteristics of verified method invocations, and effectiveness in fault 
detection. Our analysis of 4,652 test cases from 11 popular Java projects 
reveals that mock assertions are primarily applied to validate method calls 
involving external resources or reflecting code path traversal. We find that 
mock assertions complement traditional test assertions by ensuring desired 
side effects, validating control flow, and checking internal computation 
results. Our findings provide a foundation for future research such as 
automated test generation that supports mock assertions.

While unit testing focuses on isolating and emulating interactions at the 
component level, system testing requires emulating interactions between the 
system under test and external systems. A prominent example is code 
intelligence tools, where the interaction between code editors and language 
servers is central. At the core is the Language Server Protocol (LSP), which 
standardizes communication to provide code intelligence features. The 
reliability of LSP servers is a growing concern, as crashes can disrupt all 
code intelligence features. Despite widespread adoption, no existing 
techniques specifically target LSP server testing. To bridge this gap, we 
present LSPFuzz, a grey-box hybrid fuzzer that emulates code editor behaviors 
for systematic LSP server testing. Effective LSP server testing requires 
holistic mutation of source code and editor operations, as bugs often 
manifest from their combinations. We employ a two-stage mutation pipeline: 
syntax-aware mutations to source code, followed by context-aware dispatching 
of editor operations. We evaluated LSPFuzz on four widely used LSP servers. 
LSPFuzz demonstrated superior performance compared to binary and 
grammar-based fuzzers, and uncovered previously unknown bugs in real-world 
LSP servers. Our work advances the quality assurance of LSP servers and 
provides foundational insights for future research.

In summary, this thesis addresses the multifaceted challenges of emulating 
interactions in software testing across both unit and system levels. These 
contributions not only provide practical solutions for developers but also 
lay the groundwork for future research in interaction emulation for effective 
software testing. As software systems continue to grow in complexity, the 
approaches and insights presented in this thesis will be instrumental in 
ensuring their correctness and robustness.


Date:                   Thursday, 31 July 2025

Time:                   4:00pm - 6:00pm

Venue:                  Room 3494
                        Lifts 25/26

Committee Members:      Prof. Shing-Chi Cheung (Supervisor)
                        Dr. Shuai Wang (Chairperson)
                        Dr. Lionel Parreaux