Systematic and Lightweight Techniques to Preempt Flaky Tests

Speaker: Dr Wing Lam
         George Mason University

Title: "Systematic and Lightweight Techniques to Preempt Flaky Tests"

Date:   Monday, 29 May 2023

Time:   11:00am - 12 noon

Venue:  Room 5566 (via lift 27/28), HKUST

Abstract:

As software evolves, developers typically perform regression testing to
ensure that their code changes do not break existing functionality. During
regression testing, developers often waste time debugging their code
changes because of spurious failures from flaky tests, which are tests
that nondeterministically pass or fail on the same code. These spurious
failures mislead developers because the failures are due to bugs that
existed before the code changes. My work on characterizing flaky tests has
helped open the research topic of flaky tests, and many companies (e.g.,
Facebook, Google, Microsoft) have since highlighted flaky tests as a major
challenge in their software development.


In this talk, I will describe my recent work on proactively detecting,
i.e., preempting, flaky tests. A prominent kind of flaky tests is
order-dependent (OD) flaky tests, which pass when run in one test order
but fail when run in a different test order. To exhaustively detect OD
tests, one may need to run a prohibitively expensive number of test
orders, e.g., N! test orders, where N is the number of tests. To help with
this problem, my results include automated techniques to detect and
partially classify flaky tests via (1) random-based and systematic-based
repeated executions of the test suite and (2) the detection of
non-idempotent-outcome tests, which are tests that may pass in the first
run but fail in the second when the test is run repeatedly. Overall, my
work has helped detect more than 5000 flaky tests in over 300 open-source
projects.


*******************
Biography:

Dr Wing Lam is an Assistant Professor in the Computer Science department
at George Mason University. He is also a Part-Time Applied Research
Scientist at Fraunhofer CMA. Dr. Lam works on several topics in software
engineering, with a focus on software testing. His research improves
software dependability by characterizing bugs and developing novel
techniques to detect and tame bugs. Wing has published in top-tier
conferences such as ESEC/FSE, ICSE, ISSTA, OOPSLA, and TACAS. His
techniques have helped detect and fix bugs in open-source projects and
have impacted how Dragon Testing, Microsoft, and Tencent developers test
their code. Wing has been the recipient of several awards, including an
ACM SIGSOFT Outstanding Doctoral Dissertation Award. More information is
available on his web page: https://cs.gmu.edu/~winglam