SBEST: Spectrum-Based Fault Localization Without Fault-Triggering Tests

TL;DR

This paper addresses the challenge of fault localization when fault-triggering tests are unavailable by leveraging crash reports and their stack traces. It introduces SBEST, a Spectrum-Based Fault Localization method augmented with stack-trace information, and demonstrates its effectiveness on 60 crash-report bugs from Defects4J v2.0, where fault-triggering tests are rare. The study shows that stack traces are highly informative for locating buggy methods (average distance 0.34) and that SBEST achieves substantial performance gains (MAP ~0.4285, MRR ~0.4965) over SBFL baselines lacking failing tests. The work highlights the practical value of runtime signals in debugging and provides publicly accessible data to enable replication and further research in stack-trace–assisted fault localization and related tasks.

Abstract

Fault localization is a critical step in software maintenance. Yet, many existing techniques, such as Spectrum-Based Fault Localization (SBFL), rely heavily on the availability of fault-triggering tests to be effective. In practice, especially for crash-related bugs, such tests are frequently unavailable. Meanwhile, bug reports containing stack traces often serve as the only available evidence of runtime failures and provide valuable context for debugging. This study investigates the feasibility of using stack traces from crash reports as proxies for fault-triggering tests in SBFL. Our empirical analysis of 60 crash-report bugs in Defects4J reveals that only 3.33% of these bugs have fault-triggering tests available at the time of the bug report creation. However, 98.3% of bug fixes directly address the exception observed in the stack trace, and 78.3% of buggy methods are reachable within an average of 0.34 method calls from the stack trace. These findings underscore the diagnostic value of stack traces in the absence of failing tests. Motivated by these findings, we propose SBEST, a novel approach that integrates stack trace information with test coverage data to perform fault localization when fault-triggering tests are missing. SBEST shows an improvement, with a 32.22% increase in Mean Average Precision (MAP) and a 17.43% increase in Mean Reciprocal Rank (MRR) compared to baseline approaches under the scenario where fault-triggering tests are absent.

Figures (6)

• Figure 1: Example bug report (COMPRESS-181). The bug report addresses a problem in reading TAR files that contain symlinks, which cause an IOException to be thrown. The report contains the stack trace of the error (gray box), and an example of TAR file (simple-aix-native-tar.tar) that triggers the bug is attached. When this bug report was created, there were no failing tests.
• Figure 2: An example of the structure of stack trace entries in Java.
• Figure 3: Extract from the bugfix patch for Cli-5, classified in the Exception Prevention category.
• Figure 4: Extract from the bugfix patch for Closure-152, classified in the Exception Conversion category.
• Figure 5: Extract from the bugfix patch for COMPRESS-12, classified in the Exception Wrapping category.