Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamic Test Case Prioritization in Industrial Test Result Datasets

Alina Torbunova, Per Erik Strandberg, Ivan Porres

TL;DR

This work tackles accelerating fault detection in continuous integration by augmenting static test case prioritization with a dynamic scheduling layer based on conditional probability of co-failures. The approach reorders pending tests on the fly during CI cycles using pairwise test correlations derived from historical data, independently of the static baseline. Evaluated on three industrial datasets (Westermo, Paint Control, IOF/ROL) using APFD, the method yields meaningful improvements over suboptimal static orders, particularly for Random and Worst baselines, while potentially reducing performance when the static order is near-optimal. While effective, the method relies on fixed history length and a fixed update parameter, suggesting future work to adapt these settings per system and integrate with more baselines.

Abstract

Regression testing in software development checks if new software features affect existing ones. Regression testing is a key task in continuous development and integration, where software is built in small increments and new features are integrated as soon as possible. It is therefore important that developers are notified about possible faults quickly. In this article, we propose a test case prioritization schema that combines the use of a static and a dynamic prioritization algorithm. The dynamic prioritization algorithm rearranges the order of execution of tests on the fly, while the tests are being executed. We propose to use a conditional probability dynamic algorithm for this. We evaluate our solution on three industrial datasets and utilize Average Percentage of Fault Detection for that. The main findings are that our dynamic prioritization algorithm can: a) be applied with any static algorithm that assigns a priority score to each test case b) can improve the performance of the static algorithm if there are failure correlations between test cases c) can also reduce the performance of the static algorithm, but only when the static scheduling is performed at a near optimal level.

Dynamic Test Case Prioritization in Industrial Test Result Datasets

TL;DR

This work tackles accelerating fault detection in continuous integration by augmenting static test case prioritization with a dynamic scheduling layer based on conditional probability of co-failures. The approach reorders pending tests on the fly during CI cycles using pairwise test correlations derived from historical data, independently of the static baseline. Evaluated on three industrial datasets (Westermo, Paint Control, IOF/ROL) using APFD, the method yields meaningful improvements over suboptimal static orders, particularly for Random and Worst baselines, while potentially reducing performance when the static order is near-optimal. While effective, the method relies on fixed history length and a fixed update parameter, suggesting future work to adapt these settings per system and integrate with more baselines.

Abstract

Regression testing in software development checks if new software features affect existing ones. Regression testing is a key task in continuous development and integration, where software is built in small increments and new features are integrated as soon as possible. It is therefore important that developers are notified about possible faults quickly. In this article, we propose a test case prioritization schema that combines the use of a static and a dynamic prioritization algorithm. The dynamic prioritization algorithm rearranges the order of execution of tests on the fly, while the tests are being executed. We propose to use a conditional probability dynamic algorithm for this. We evaluate our solution on three industrial datasets and utilize Average Percentage of Fault Detection for that. The main findings are that our dynamic prioritization algorithm can: a) be applied with any static algorithm that assigns a priority score to each test case b) can improve the performance of the static algorithm if there are failure correlations between test cases c) can also reduce the performance of the static algorithm, but only when the static scheduling is performed at a near optimal level.
Paper Structure (10 sections, 1 figure, 1 table, 1 algorithm)

This paper contains 10 sections, 1 figure, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Previous Work on Test Case Prioritization
  3. Combined Static and Dynamic Test Prioritization
  4. Dynamic Scheduling using Conditional Probability
  5. Evaluation
  6. General description of the datasets
  7. The Westermo dataset
  8. The Paint Control and the IOF/ROL datasets
  9. Results
  10. Discussion and Conclusions

Figures (1)

  • Figure 1: Comparison for APFD for static and dynamic prioritization algorithms (higher values are better). The first nine systems are from the Westermo dataset, the last two are from Paint Control and IOF/ROL.