LLM-based Failing Test Generation

PhD Thesis Proposal Defence


Title: "LLM-based Failing Test Generation"

by

Mr. Tsz On LI


Abstract:

A central objective of software testing is to generate failing tests. However, 
automatically generating such test cases is challenging in practice, as it 
requires automatic construction of test oracles (i.e., criteria for 
determining whether an observed program behavior is correct); oracle 
construction is undecidable in general. Large language models (LLMs) offer a 
new opportunity for addressing these challenges. In this thesis, we conduct 
the first study to investigate the use of ChatGPT in finding failing tests for 
a given program. Our study shows that ChatGPT achieves a low success rate of 
28.8%, and identifies three key limitations of LLMs: (1) its insensitivity to 
the subtle differences ("nuances") between buggy and correct code, causing it 
to often consider buggy code as correct; (2) its performance further degrades 
as code complexity increases; and (3) it can be susceptible to training bias 
and stochasticity, causing it to return incorrect failing tests.

This thesis develops three techniques that address these limitations in turn. 
To address (1), our insight is that LLMs' weakness can be turned into 
strengths: because LLMs overlook the subtle differences between buggy and 
correct code, they can infer the intended functionality even of a buggy 
program. Based on this insight, we implement Differential Prompting, the first 
failing test generation paradigm. It first leverages an LLM to infer the 
intended functionalities of a program-under-test (PUT). Then, it generates 
alternative implementations of the faulty program based on the inferred 
intended functionalities. Finally, it derives tests that expose 
inconsistencies between the original program and alternative implementations. 
Such an inconsistency is reported as a candidate failing test.

To address limitation (2), we are inspired by an earlier finding that 
constituent snippets within a program typically implement simpler coding tasks 
compared to the program as a whole. As a result, LLMs can be leveraged to 
generate failing tests that target a program's constituent snippets, thereby 
revealing the program defects. To address limitation (3), our idea is that 
LLMs' training bias and stochasticity can be mitigated through calibrating 
LLMs' chain-of-thought reasoning in the inference phase. Specifically, we 
observe that outputs consistently returned by both the PUT and all alternative 
implementations are likely to be the correct outputs. This is because such 
outputs are generated by independent efforts (i.e., human developers and 
LLMs). These outputs can be augmented (e.g., through mutation) to form a 
calibration dataset that identifies the training biases and stochasticity of 
the LLM's inference and supports calibrating its reasoning accordingly.

Together, these techniques advance the state of the art in LLM-based failing 
test generation. We evaluate them on several benchmarks and real-world 
subjects, and discuss limitations and future directions.


Date:                   Wednesday, 23 September 2026

Time:                   9:00am - 11:00am

Venue:                  Room 3494
                        Lift 25/26

Committee Members:      Prof. Shing-Chi Cheung (Supervisor)
                        Prof. Raymond Wong (Chairperson)
                        Prof. Kai Chen
                        Dr. Xiaomin Ouyang