Evaluating Developer-written Unit Test Case Reduction for Java -- A Replication Study
Tuan D Le, Brandon Wilber, Arpit Christi
TL;DR
The paper addresses whether leaf vs non-leaf statements in failing unit tests exhibit different reduction patterns, by replicating a C# study in the Java ecosystem using the tool ReduJavator. It applies the same experimental protocol to 30 Defects4J bugs and 30 mutants across six projects, measuring absolute and relative reductions for TreeStmt and NonTreeStmt categories with ARS,PRS,ATRS, PTRS, ANTRS, PNTRS and using nonparametric statistics. The main findings show leaf/non-tree statements dominate reductions by about nine times compared to tree statements, while the evidence for a higher removal probability for non-tree statements is inconclusive. The work underscores the value of replication for assessing generalizability and provides Java-based tooling and datasets to support future reproducibility efforts in software testing research.
Abstract
Abstract: Failing test case reduction can promote efficient debugging because a developer may not need to observe components that are not relevant to inducing failure. Failing test case reduction can also improve the efficiency of fault localization. These considerations have prompted researchers to study the reduction process, the reduction output, and the removed entities. Christi et al. studied test reduction using a tool called ReduSharptor for C# tests. They considered the test to be an Abstract Syntax Tree (AST). Based on that, they studied the reduction outcome and removed entities in terms of Leaf nodes and Non-Leaf nodes of the AST. They claimed that (1) leaf nodes are removed in large numbers, and (2) the probability of removal is slightly higher than non-leaf nodes. We replicate their results using a different test case reduction tool, ReduJavator, for Java unit tests. We evaluate test reduction using 30 randomly chosen bugs from the Defects4J database and 30 mutants for 6 open-source projects. Our results confirm their first claim: leaf nodes are removed in large numbers. Our results are inconclusive regarding their second claim; we cannot confirm that the probability of removal is higher for non-leaf nodes.