MulTi-Wise Sampling: Trading Uniform T-Wise Feature Interaction Coverage for Smaller Samples

TL;DR

MulTi-Wise Sampling tackles the challenge of expensive high-$t$-wise coverage in testing highly configurable systems by introducing feature-group based prioritization. It partitions features into groups with assigned $t$ values and generates group-specific samples using an iterative process that leverages the YasA sampler for intermediate results. The approach reduces sample size and generation time at the cost of not achieving uniform full coverage across all features, showing comparable performance to state-of-the-art baselines when feature criticality is known. This work provides a practical method for resource-constrained testing in software product lines, with demonstrated applicability to real-world subject systems and a pathway for integrating criticality-aware sampling into existing pipelines.

Abstract

Ensuring the functional safety of highly configurable systems often requires testing representative subsets of all possible configurations to reduce testing effort and save resources. The ratio of covered t-wise feature interactions (i.e., T-Wise Feature Interaction Coverage) is a common criterion for determining whether a subset of configurations is representative and capable of finding faults. Existing t-wise sampling algorithms uniformly cover t-wise feature interactions for all features, resulting in lengthy execution times and large sample sizes, particularly when large t-wise feature interactions are considered (i.e., high values of t). In this paper, we introduce a novel approach to t-wise feature interaction sampling, questioning the necessity of uniform coverage across all t-wise feature interactions, called \emph{\mulTiWise{}}. Our approach prioritizes between subsets of critical and non-critical features, considering higher t-values for subsets of critical features when generating a t-wise feature interaction sample. We evaluate our approach using subject systems from real-world applications, including \busybox{}, \soletta{}, \fiasco{}, and \uclibc{}. Our results show that sacrificing uniform t-wise feature interaction coverage between all features reduces the time needed to generate a sample and the resulting sample size. Hence, \mulTiWise{} Sampling offers an alternative to existing approaches if knowledge about feature criticality is available.

Figures (5)

• Figure 1: Feature diagram of a simplified car system consisting of 11 features. The feature boxes in the diagram show the feature's name and its literal representation in parentheses.
• Figure 2: Results of measuring pair-wise ($t = 2$) coverage.
• Figure 3: Results of measuring three-wise ($t = 3$) coverage.
• Figure 4: Results of measuring sampling time.
• Figure 5: Results of measuring the sample size.