Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Large-Scale Study of Call Graph-based Impact Prediction using Mutation Testing

Vincenzo Musco, Martin Monperrus, Philippe Preux

TL;DR

An evaluation technique is proposed to predict impact propagation based on 17,000 mutants and shows that graph sophistication increases the completeness of impact prediction, but surprisingly, the most basic call graph gives the best trade-off between precision and recall for impact prediction.

Abstract

In software engineering, impact analysis involves predicting the software elements (e.g., modules, classes, methods) potentially impacted by a change in the source code. Impact analysis is required to optimize the testing effort. In this paper, we propose an evaluation technique to predict impact propagation. Based on 10 open-source Java projects and 5 classical mutation operators, we create 17,000 mutants and study how the error they introduce propagates. This evaluation technique enables us to analyze impact prediction based on four types of call graph. Our results show that graph sophistication increases the completeness of impact prediction. However, and surprisingly to us, the most basic call graph gives the best trade-off between precision and recall for impact prediction.

A Large-Scale Study of Call Graph-based Impact Prediction using Mutation Testing

TL;DR

An evaluation technique is proposed to predict impact propagation based on 17,000 mutants and shows that graph sophistication increases the completeness of impact prediction, but surprisingly, the most basic call graph gives the best trade-off between precision and recall for impact prediction.

Abstract

In software engineering, impact analysis involves predicting the software elements (e.g., modules, classes, methods) potentially impacted by a change in the source code. Impact analysis is required to optimize the testing effort. In this paper, we propose an evaluation technique to predict impact propagation. Based on 10 open-source Java projects and 5 classical mutation operators, we create 17,000 mutants and study how the error they introduce propagates. This evaluation technique enables us to analyze impact prediction based on four types of call graph. Our results show that graph sophistication increases the completeness of impact prediction. However, and surprisingly to us, the most basic call graph gives the best trade-off between precision and recall for impact prediction.

Paper Structure

This paper contains 21 sections, 7 equations, 3 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Definitions
  4. A Novel Evaluation Technique for CIA
  5. Impact Evaluation with Mutants
  6. One-Impact Mutant-Level Accuracy Metrics
  7. Global Accuracy Metrics
  8. Call Graphs for Impact Prediction
  9. Elaborated Example
  10. Visualization
  11. Implementation
  12. Experimental Evaluation
  13. Dataset
  14. Results
  15. Discussion
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Simple Java source code and the four types of graphs obtained from it: $\mathscr{C}_{S}$, $\mathscr{C}_{B}$, $\mathscr{C}_{H}$ and $\mathscr{C}_{F}$.
  • Figure 2: Example of a call graph in which a change has been introduced. The class graph includes application nodes, test nodes and call edges. The rectangles illustrate Bohner's sets.
  • Figure 3: Effect of a particular mutation in the Apache Commons Lang project. Only the interesting part of the call graph is represented here; the call graph is much larger, made of 6000+ nodes. Black circle are the nodes that propagate the mutation injected in the node denoted with a blue cross. Nodes illustrated by green boxes, red diamonds and purple stars are test cases related to the injected mutation. Green boxes nodes are test cases that are correctly predicted as impacted by the injected mutation; these are true positives. Red diamonds nodes are test cases that are predicted as impacted, but are not; these are false positives. Purple stars nodes are test cases that should have been predicted as being impacted but have not been; these are false negatives.