Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generating Maximal Configurations and Their Variants Using Code Metrics

Tuba Yavuz, Chin Khor, Ken, Bai, Robyn Lutz

TL;DR

This paper addresses the challenge of generating test configurations for highly configurable systems by combining semantic (presence conditions) and code-metric based approaches. It introduces CONFIZZ, a framework that uses MaxSAT-based maximal configuration generation and code-metric guided fuzzing to produce both maximal configurations and minimal coverage-focused sets, guided by metrics such as logical code unit (LCU) coverage. On BusyBox, MaxSAT-based generation improved code-coverage metrics and revealed bug-relevant configurations, while presence-condition fuzzing offered strong performance when multiple high-score configurations were needed. The results demonstrate that integrating constraint solving with code metrics yields more effective and scalable configuration testing, with practical benefits for bug finding and test prioritization.

Abstract

Testing configurable systems continues to be challenging and costly. Generation of configurations for testing tends to use either techniques based on semantic sampling (e.g., logical formulas over configuration variables, often called presence conditions) or structural code metrics (e.g., code coverage). In this paper we describe our hybrid approaches that combine these two kinds of techniques to good effect. We present new configuration-generation algorithms that leverage constraint solving (SAT and MaxSAT) and configuration fuzzing, and implement our approach in a configuration-generation framework, CONFIZZ. CONFIZZ both enables the generation of maximal configurations (maximal sets of presence conditions that can be satisfied together) and performs code-metric guided configuration fuzzing. Results from evaluation on BusyBox, a highly configurable benchmark, show that our MaxSAT-based configuration generation achieves better coverage for several code metrics. Results also show that, when high coverage of multiple configurations is needed, CONFIZZ's presence-condition fuzzing outperforms alternatives.

Generating Maximal Configurations and Their Variants Using Code Metrics

TL;DR

This paper addresses the challenge of generating test configurations for highly configurable systems by combining semantic (presence conditions) and code-metric based approaches. It introduces CONFIZZ, a framework that uses MaxSAT-based maximal configuration generation and code-metric guided fuzzing to produce both maximal configurations and minimal coverage-focused sets, guided by metrics such as logical code unit (LCU) coverage. On BusyBox, MaxSAT-based generation improved code-coverage metrics and revealed bug-relevant configurations, while presence-condition fuzzing offered strong performance when multiple high-score configurations were needed. The results demonstrate that integrating constraint solving with code metrics yields more effective and scalable configuration testing, with practical benefits for bug finding and test prioritization.

Abstract

Testing configurable systems continues to be challenging and costly. Generation of configurations for testing tends to use either techniques based on semantic sampling (e.g., logical formulas over configuration variables, often called presence conditions) or structural code metrics (e.g., code coverage). In this paper we describe our hybrid approaches that combine these two kinds of techniques to good effect. We present new configuration-generation algorithms that leverage constraint solving (SAT and MaxSAT) and configuration fuzzing, and implement our approach in a configuration-generation framework, CONFIZZ. CONFIZZ both enables the generation of maximal configurations (maximal sets of presence conditions that can be satisfied together) and performs code-metric guided configuration fuzzing. Results from evaluation on BusyBox, a highly configurable benchmark, show that our MaxSAT-based configuration generation achieves better coverage for several code metrics. Results also show that, when high coverage of multiple configurations is needed, CONFIZZ's presence-condition fuzzing outperforms alternatives.
Paper Structure (15 sections, 5 equations, 12 figures, 8 tables, 3 algorithms)

This paper contains 15 sections, 5 equations, 12 figures, 8 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Approach
  4. Code Metrics and Presence Conditions
  5. Generating a Maximal Configuration
  6. Generating a Minimal Set of Configurations
  7. Generating Maximal Variants
  8. Configuration Fuzzing
  9. Evaluation
  10. RQ1 on the Default Configuration
  11. RQ2 on Full Coverage
  12. RQ3 on Comparison of Approaches
  13. RQ4 on Bug Relevance of Code Metrics
  14. Conclusions
  15. Acknowledgements

Figures (12)

  • Figure 1: Various configuration generation approaches implemented in CONFIZZ are shown with thick rounded rectangles. Solid arrows denote data flow and dashed arrows denote control flow. C and CS denote a single configuration and a set of configurations, respectively.
  • Figure 2: Example code with various presence conditions.
  • Figure 3: Maximum LCU score achieved by Presence Condition Fuzzing (PCF) for various number of fuzzing cycles for the editors components.
  • Figure 4: Maximum LCU score achieved by Presence Condition Fuzzing (PCF) for various number of fuzzing cycles for the coreutils components.
  • Figure 5: Comparison of methods w.r.t. LCU max score for awk-ed. PCF (cycles=25). CVF (cycles=5).
  • ...and 7 more figures