Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Causal Inference for the Effect of Code Coverage on Bug Introduction

Lukas Schulte, Gordon Fraser, Steffen Herbold

TL;DR

This work tackles whether code coverage causally reduces bug introduction in mature JavaScript/TypeScript projects. It adopts a formal causal framework using a DAG to identify confounders, GPS for continuous exposure, and doubly robust IPW-based estimation to derive the $ATE$ and a dose–response curve, aiming to reveal non-linear effects and diminishing returns. A pre-registered protocol introduces a new longitudinal dataset combining coverage, code metrics, issues, and changes, and specifies rigorous data-mining and estimation steps to move beyond correlational findings. The study's outcomes have practical implications for QA policy by clarifying how much coverage matters and where resources should be allocated in testing practices.

Abstract

Context: Code coverage is widely used as a software quality assurance measure. However, its effect, and specifically the advisable dose, are disputed in both the research and engineering communities. Prior work reports only correlational associations, leaving results vulnerable to confounding factors. Objective: We aim to quantify the causal effect of code coverage (exposure) on bug introduction (outcome) in the context of mature JavaScript and TypeScript open source projects, addressing both the overall effect and its variance across coverage levels. Method: We construct a causal directed acyclic graph to identify confounders within the software engineering process, modeling key variables from the source code, issue- and review systems, and continuous integration. Using generalized propensity score adjustment, we will apply doubly robust regression-based causal inference for continuous exposure to a novel dataset of bug-introducing and non-bug-introducing changes. We estimate the average treatment effect and dose-response relationship to examine potential non-linear patterns (e.g., thresholds or diminishing returns) within the projects of our dataset.

Causal Inference for the Effect of Code Coverage on Bug Introduction

TL;DR

This work tackles whether code coverage causally reduces bug introduction in mature JavaScript/TypeScript projects. It adopts a formal causal framework using a DAG to identify confounders, GPS for continuous exposure, and doubly robust IPW-based estimation to derive the and a dose–response curve, aiming to reveal non-linear effects and diminishing returns. A pre-registered protocol introduces a new longitudinal dataset combining coverage, code metrics, issues, and changes, and specifies rigorous data-mining and estimation steps to move beyond correlational findings. The study's outcomes have practical implications for QA policy by clarifying how much coverage matters and where resources should be allocated in testing practices.

Abstract

Context: Code coverage is widely used as a software quality assurance measure. However, its effect, and specifically the advisable dose, are disputed in both the research and engineering communities. Prior work reports only correlational associations, leaving results vulnerable to confounding factors. Objective: We aim to quantify the causal effect of code coverage (exposure) on bug introduction (outcome) in the context of mature JavaScript and TypeScript open source projects, addressing both the overall effect and its variance across coverage levels. Method: We construct a causal directed acyclic graph to identify confounders within the software engineering process, modeling key variables from the source code, issue- and review systems, and continuous integration. Using generalized propensity score adjustment, we will apply doubly robust regression-based causal inference for continuous exposure to a novel dataset of bug-introducing and non-bug-introducing changes. We estimate the average treatment effect and dose-response relationship to examine potential non-linear patterns (e.g., thresholds or diminishing returns) within the projects of our dataset.
Paper Structure (14 sections, 2 figures, 1 table)

This paper contains 14 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Causal Analysis in Software Engineering Research
  4. Code Coverage and Bug Introduction
  5. Research Questions and Hypotheses
  6. Research Protocol
  7. Subjects
  8. Variables
  9. Relationships
  10. Execution Plan
  11. Limitations
  12. Conclusion
  13. Future Work
  14. Publication of Data

Figures (2)

  • Figure 1: Visual representation of commits considered in the analysis. Feature-branch commits (FBC) are potential bug-introducing commits, while merge commits (MC) integrate the changes from the FBC into the main branch.
  • Figure 2: The proposed causal graph of (latent) variables. Effect between code coverage and bug introduction highlighted in red.