Understanding Bug-Reproducing Tests: A First Empirical Study
Andre Hora, Gordon Fraser
TL;DR
Understanding Bug-Reproducing Tests presents an initial empirical study of 642 bug-reproducing tests from 15 real-world Python systems to determine whether these tests differ from regular tests and how bugs map to tests. The authors find that bug-reproducing tests are largely similar to ordinary tests in LOC, assertions, and complexity, with modest elevations in try/except usage and weak assertions. They also show that 95% of bug-reproducing tests reproduce a single bug, while 5% reproduce multiple bugs, and that some bugs are tested by multiple tests or across tests. The work discusses practical implications for improving bug-reproducing tests, such as promoting strong assertions and test reduction, and outlines directions for future research in semantic properties and automation.
Abstract
Developers create bug-reproducing tests that support debugging by failing as long as the bug is present, and passing once the bug has been fixed. These tests are usually integrated into existing test suites and executed regularly alongside all other tests to ensure that future regressions are caught. Despite this co-existence with other types of tests, the properties of bug-reproducing tests are scarcely researched, and it remains unclear whether they differ fundamentally. In this short paper, we provide an initial empirical study to understand bug-reproducing tests better. We analyze 642 bug-reproducing tests of 15 real-world Python systems. Overall, we find that bug-reproducing tests are not (statistically significantly) different from other tests regarding LOC, number of assertions, and complexity. However, bug-reproducing tests contain slightly more try/except blocks and ``weak assertions'' (e.g.,~\texttt{assertNotEqual}). Lastly, we detect that the majority (95%) of the bug-reproducing tests reproduce a single bug, while 5% reproduce multiple bugs. We conclude by discussing implications and future research directions.