Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SeqTG: Scalable Combinatorial Test Generation via Sequential Integer Linear Programming

Sitong Yang, Wanying Bao, Yinyin Song, Yueting Cheng, Qian Li, Chao Wei

Abstract

Combinatorial Testing (CT) is essential for detecting interaction-triggered faults, yet generating minimal Covering Arrays under complex constraints remains an unresolved NP-hard challenge. Current greedy algorithms are highly scalable but suffer from severe ``diminishing returns'': they efficiently cover initial interactions but produce bloated, redundant test suites when struggling to pack the final few difficult pairs. While exact mathematical programming could theoretically address this inefficiency, it has historically been intractable due to combinatorial explosion. In this paper, we pioneer the application of exact mathematical modeling to CT by introducing SeqTG, a scalable framework based on Sequential Integer Linear Programming (ILP). To circumvent the scalability barrier, SeqTG employs a novel Warm-Start strategy: a rapid greedy initialization first clears the ``easy'' interactions, allowing the rigorous ILP solver to exclusively optimize the fragmented, difficult-to-cover remainder. The pipeline operates in three stages: (1) a Constraint-First phase grouping must-include requirements via graph partitioning; (2) an Incremental Optimization phase targeting the remaining interactions with sequential ILP; and (3) a Global Minimization phase eliminating redundancies via set-covering. Extensive evaluations across standard benchmarks and 200 large-scale configurations validate the framework's efficacy. The results demonstrate that SeqTG effectively eradicates late-stage bloat, achieving state-of-the-art test suite compactness and strict constraint adherence.

SeqTG: Scalable Combinatorial Test Generation via Sequential Integer Linear Programming

Abstract

Combinatorial Testing (CT) is essential for detecting interaction-triggered faults, yet generating minimal Covering Arrays under complex constraints remains an unresolved NP-hard challenge. Current greedy algorithms are highly scalable but suffer from severe ``diminishing returns'': they efficiently cover initial interactions but produce bloated, redundant test suites when struggling to pack the final few difficult pairs. While exact mathematical programming could theoretically address this inefficiency, it has historically been intractable due to combinatorial explosion. In this paper, we pioneer the application of exact mathematical modeling to CT by introducing SeqTG, a scalable framework based on Sequential Integer Linear Programming (ILP). To circumvent the scalability barrier, SeqTG employs a novel Warm-Start strategy: a rapid greedy initialization first clears the ``easy'' interactions, allowing the rigorous ILP solver to exclusively optimize the fragmented, difficult-to-cover remainder. The pipeline operates in three stages: (1) a Constraint-First phase grouping must-include requirements via graph partitioning; (2) an Incremental Optimization phase targeting the remaining interactions with sequential ILP; and (3) a Global Minimization phase eliminating redundancies via set-covering. Extensive evaluations across standard benchmarks and 200 large-scale configurations validate the framework's efficacy. The results demonstrate that SeqTG effectively eradicates late-stage bloat, achieving state-of-the-art test suite compactness and strict constraint adherence.
Paper Structure (17 sections, 6 equations, 3 figures, 5 tables, 2 algorithms)

This paper contains 17 sections, 6 equations, 3 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. The SeqTG Framework
  3. Monolithic Optimization Approaches
  4. ILP Formulation for Maximizing Coverage
  5. The Scalability Barrier
  6. From Monolithic to Sequential Optimization
  7. Constraint Handling in the Sequential Model
  8. Global Test Suite Minimization via Set Covering
  9. The Complete SeqTG Algorithm
  10. Experiments and Results
  11. Industrial Case Study: 5G Baseband Configuration Testing
  12. Comparison on Standard Benchmarks and Weight Ablation
  13. Performance on Large-Scale Unconstrained Problems
  14. Performance on Large-Scale Constrained Problems
  15. Computational Overhead and Warm-Start Efficiency
  16. ...and 2 more sections

Figures (3)

  • Figure 1: The efficiency curve and diminishing returns of greedy combinatorial testing algorithms. The data is derived from 100 independent runs of PICT and Jenny on randomly generated system configurations (with the number of factors and levels uniformly ranging from 5 to 30). The left panel illustrates the mean pairwise coverage growth and its variance, showing rapid initial coverage. The right panel quantifies the inefficiency in the later stages, demonstrating that covering the final 10% of interactions (90-100%) disproportionately consumes roughly 40% of the total generated test cases.
  • Figure 2: Comprehensive performance comparison of SeqTG against IPO, AllPairs, PICT, and Jenny across 100 unconstrained test configurations. (a) Average test suite size with standard deviation. (b) Box plots showing the percentage reduction in test suite size achieved by SeqTG relative to other methods. (c) Performance profiles illustrating the fraction of problems solved within a given performance ratio. (d) Stacked bar chart showing the ranking distribution for each method (Rank 1 is best).
  • Figure 3: Comprehensive performance comparison of SeqTG against IPO, AllPairs, PICT, and Jenny across 100 constrained test configurations. (a) Average test suite size with standard deviation. (b) Box plots showing the percentage reduction in test suite size achieved by SeqTG relative to other methods. (c) Performance profiles illustrating the fraction of problems solved within a given performance ratio. (d) Stacked bar chart showing the ranking distribution for each method (Rank 1 is best).