Taming Flaky Tests in a Non-Deterministic World

Speaker:        Wing LAM
                University of Illinois at Urbana-Champaign

Title:          "Taming Flaky Tests in a Non-Deterministic World"

Date:           Wednesday, 17 February 2021

Time:           11am - 12 noon

Zoom Meeting:
https://hkust.zoom.us/j/465698645?pwd=c2E4VTE3b2lEYnBXcyt4VXJITXRIdz09

Meeting ID:     465-698-645
Passcode:       20202021


Abstract:

As software evolves, developers typically perform regression testing to
ensure that their code changes do not break existing functionality. During
regression testing, developers often waste time debugging their code
changes because of spurious failures from flaky tests, which are tests
that nondeterministically pass or fail on the same code. These spurious
failures mislead developers because the failures are due to bugs that
existed before the code changes. My work on characterizing flaky tests has
helped open the research topic of flaky tests, and many companies (e.g.,
Facebook, Google, Microsoft) have since highlighted flaky tests as a major
challenge in their software development.

In this talk, I will describe my recent work on taming flaky tests. Two
prominent kinds of flaky tests are order-dependent flaky tests, which pass
when run in one order but fail when run in a different order, and
async-wait flaky tests, which pass if an asynchronous call finishes on
time but fail if it finishes too late. My results include the first
automated techniques to (1) fix order-dependent flaky tests, fixing 92% of
such flaky tests in a public dataset; (2) reduce the number of spurious
failures from order-dependent flaky tests, reducing such failures by 73%;
and (3) speed up async-wait flaky tests while also reducing their spurious
failures, speeding up such tests by 38%. Overall, my work has helped
detect more than 2000 flaky tests and fix more than 500 flaky tests in
over 150 open-source projects.


****************
Biography:

Wing Lam is a PhD candidate in the Computer Science department at the
University of Illinois at Urbana-Champaign where he is co-advised by
Professors Tao Xie and Darko Marinov. He works on several topics in
software engineering, with a focus on software testing. Wing's research
improves software dependability by characterizing bugs and developing
novel techniques to detect and tame bugs. He has published in top-tier
conferences in software engineering (ESEC/FSE, ICSE), software testing
(ICST, ISSTA), and beyond (OOPSLA, TACAS). His techniques have helped
detect and fix bugs in open-source projects and have impacted how
Microsoft and Tencent developers test their code. Wing has been awarded
several fellowships and scholarships, including a Google - CMD-IT
Dissertation Fellowship Award. More information is available on his web
page: http://mir.cs.illinois.edu/winglam