Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Simple Fault Localization using Execution Traces

Julian Aron Prenner, Romain Robbes

TL;DR

This work addresses the limited performance of traditional spectrum-based fault localization by leveraging execution traces to derive count spectra, basic control-flow, and lexical features, then contextualizing them with a sliding window. A gradient-boosted model (LightGBM) is trained on contextualized line-level features to predict buggy lines, and evaluated on RunBugRun and QuixBugs. The approach consistently outperforms standard SBFL metrics, notably with a window size of three and across multiple feature groups, while remaining GPU-free and easily integrable. The findings suggest that simple, trace-derived features can meaningfully enhance FL without resorting to heavy static analysis or deep learning. The work also highlights data quality and quantity as critical factors for further improvements and generalization to larger, real-world codebases.

Abstract

Traditional spectrum-based fault localization (SBFL) exploits differences in a program's coverage spectrum when run on passing and failing test cases. However, such runs can provide a wealth of additional information beyond mere coverage. Working with thousands of execution traces of short programs submitted to competitive programming contests and leveraging machine learning and additional runtime, control-flow and lexical features, we present simple ways to improve SBFL. We also propose a simple trick to integrate context information. Our approach outperforms SBFL formulae such as Ochiai on our evaluation set as well as QuixBugs and requires neither a GPU nor any form of advanced program analysis. Existing SBFL solutions could possibly be improved with reasonable effort by adopting some of the proposed ideas.

Simple Fault Localization using Execution Traces

TL;DR

This work addresses the limited performance of traditional spectrum-based fault localization by leveraging execution traces to derive count spectra, basic control-flow, and lexical features, then contextualizing them with a sliding window. A gradient-boosted model (LightGBM) is trained on contextualized line-level features to predict buggy lines, and evaluated on RunBugRun and QuixBugs. The approach consistently outperforms standard SBFL metrics, notably with a window size of three and across multiple feature groups, while remaining GPU-free and easily integrable. The findings suggest that simple, trace-derived features can meaningfully enhance FL without resorting to heavy static analysis or deep learning. The work also highlights data quality and quantity as critical factors for further improvements and generalization to larger, real-world codebases.

Abstract

Traditional spectrum-based fault localization (SBFL) exploits differences in a program's coverage spectrum when run on passing and failing test cases. However, such runs can provide a wealth of additional information beyond mere coverage. Working with thousands of execution traces of short programs submitted to competitive programming contests and leveraging machine learning and additional runtime, control-flow and lexical features, we present simple ways to improve SBFL. We also propose a simple trick to integrate context information. Our approach outperforms SBFL formulae such as Ochiai on our evaluation set as well as QuixBugs and requires neither a GPU nor any form of advanced program analysis. Existing SBFL solutions could possibly be improved with reasonable effort by adopting some of the proposed ideas.

Paper Structure

This paper contains 13 sections, 1 equation, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Obtaining traces
  5. Calculating line-level features
  6. Calculating feature windows
  7. Training
  8. Evaluation
  9. Results
  10. Ablation Study
  11. Discussion
  12. Limitations & Threads to Validity
  13. Conclusions

Figures (2)

  • Figure 1: MFR as a function of window size (x). The lowest MFR is obtained with a window size of three. Error bands show a 95% CI over three training runs.
  • Figure 2: MFR as a function of the training set size where 100 % uses the entire training data of roughly 22'000 samples (buggy programs). A window size is of $w = 3$ is used here. Error bands show a 95% CI over three training runs.