Efficient Incremental Code Coverage Analysis for Regression Test Suites

TL;DR

iJaCoCo is presented, the first incremental code coverage analysis technique, which speeds up code coverage analysis by executing a minimal subset of tests to update the coverage data affected by code changes.

Abstract

Code coverage analysis has been widely adopted in the continuous integration of open-source and industry software repositories to monitor the adequacy of regression test suites. However, computing code coverage can be costly, introducing significant overhead during test execution. Plus, re-collecting code coverage for the entire test suite is usually unnecessary when only a part of the coverage data is affected by code changes. While regression test selection (RTS) techniques exist to select a subset of tests whose behaviors may be affected by code changes, they are not compatible with code coverage analysis techniques -- that is, simply executing RTS-selected tests leads to incorrect code coverage results. In this paper, we present the first incremental code coverage analysis technique, which speeds up code coverage analysis by executing a minimal subset of tests to update the coverage data affected by code changes. We implement our technique in a tool dubbed iJaCoCo, which builds on Ekstazi and JaCoCo -- the state-of-the-art RTS and code coverage analysis tools for Java. We evaluate iJaCoCo on 1,122 versions from 22 open-source repositories and show that iJaCoCo can speed up code coverage analysis time by an average of 1.86x and up to 8.20x compared to JaCoCo.

Figures (29)

• Figure 1: Code change between version b1deb442 and 9fb4f47f.
• Figure 2: Tests selected by iJaCoCo.
• Figure 3: Example of using iJaCoCo on commons-lang: when computing the code coverage on version 9fb4f47f, JaCoCo executes all 149 tests and takes 29.75s, and iJaCoCo only executes 4 tests and takes 18.65s (speedup: 1.60$\times$).
• Figure 4: Workflow of incremental code coverage analysis.
• Figure 5: Example showing the necessity for incremental code coverage analysis selecting more test classes than RTS; directed edges represent "depends on" relationship.