Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalable and Accurate Test Case Prioritization in Continuous Integration Contexts

Ahmadreza Saboor Yaraghi, Mojtaba Bagherzadeh, Nafiseh Kahani, Lionel Briand

TL;DR

This work defines a data model that captures data sources and their relations in a typical CI environment and defines a comprehensive set of features that covers all features previously used by related studies, and develops methods and tools to collect the defined features.

Abstract

Continuous Integration (CI) requires efficient regression testing to ensure software quality without significantly delaying its CI builds. This warrants the need for techniques to reduce regression testing time, such as Test Case Prioritization (TCP) techniques that prioritize the execution of test cases to detect faults as early as possible. Many recent TCP studies employ various Machine Learning (ML) techniques to deal with the dynamic and complex nature of CI. However, most of them use a limited number of features for training ML models and evaluate the models on subjects for which the application of TCP makes little practical sense, due to their small regression testing time and low number of failed builds. In this work, we first define, at a conceptual level, a data model that captures data sources and their relations in a typical CI environment. Second, based on this data model, we define a comprehensive set of features that covers all features previously used by related studies. Third, we develop methods and tools to collect the defined features for 25 open-source software systems with enough failed builds and whose regression testing takes at least five minutes. Fourth, relying on the collected dataset containing a comprehensive feature set, we answer four research questions concerning data collection time, the effectiveness of ML-based TCP, the impact of the features on effectiveness, the decay of ML-based TCP models over time, and the trade-off between data collection time and the effectiveness of ML-based TCP techniques.

Scalable and Accurate Test Case Prioritization in Continuous Integration Contexts

TL;DR

This work defines a data model that captures data sources and their relations in a typical CI environment and defines a comprehensive set of features that covers all features previously used by related studies, and develops methods and tools to collect the defined features.

Abstract

Continuous Integration (CI) requires efficient regression testing to ensure software quality without significantly delaying its CI builds. This warrants the need for techniques to reduce regression testing time, such as Test Case Prioritization (TCP) techniques that prioritize the execution of test cases to detect faults as early as possible. Many recent TCP studies employ various Machine Learning (ML) techniques to deal with the dynamic and complex nature of CI. However, most of them use a limited number of features for training ML models and evaluate the models on subjects for which the application of TCP makes little practical sense, due to their small regression testing time and low number of failed builds. In this work, we first define, at a conceptual level, a data model that captures data sources and their relations in a typical CI environment. Second, based on this data model, we define a comprehensive set of features that covers all features previously used by related studies. Third, we develop methods and tools to collect the defined features for 25 open-source software systems with enough failed builds and whose regression testing takes at least five minutes. Fourth, relying on the collected dataset containing a comprehensive feature set, we answer four research questions concerning data collection time, the effectiveness of ML-based TCP, the impact of the features on effectiveness, the decay of ML-based TCP models over time, and the trade-off between data collection time and the effectiveness of ML-based TCP techniques.

Paper Structure

This paper contains 40 sections, 16 equations, 6 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Background
  3. ML-based Test Case Prioritization in CI Context
  4. Dependency and Impact Analysis of Source Code Entities
  5. Classification of Defect-fix Commits
  6. Data Model
  7. High-level class diagram
  8. Feature model
  9. Test Case Source Code Features (TES)
  10. Test Case Execution Record Features (REC)
  11. Test Case File Coverage Features (F_COV)
  12. Features of Covered Source Code by Test cases (COD_COV)
  13. Test Case Coverage Fault Detection Features (DET_COV)
  14. Validation
  15. Research Questions
  16. ...and 25 more sections

Figures (6)

  • Figure 1: ML-based TCP in CI Context
  • Figure 2: High-level Data Model of a CI Build
  • Figure 3: A comprehensive class diagram depicting our proposed feature group hierarchy (three high-level groups with nine low-level groups) and the relation between the feature groups and the metrics used for computing them.
  • Figure 4: Linear correlation graphs between two characteristics of subjects (x-axis) and the total data collection time of all feature groups as well as two individual feature groups (y-axis).
  • Figure 5: The histogram shows the relationship between the age of all test cases and their failure frequency. The x-axis shows the normalized age between 0 and 100 and the y-axis shows the number of failures of all test cases in a given age period.
  • ...and 1 more figures

Theorems & Definitions (8)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Definition 5
  • Definition 6
  • Definition 7
  • Definition 8