Unlocking the Secrets of Software Configuration Landscapes-Ruggedness, Accessibility, Escapability, and Transferability

TL;DR

We address how configurable software landscapes shape optimization by applying fitness landscape analysis to $86.92\text{M}$ configurations across $32$ workloads on three real-world systems. We construct local optima networks and quantify ruggedness, basin sizes, and escapability, showing that top local optima have large basins and inferior optima are easily escapable, with cross-workload structural similarities. We demonstrate that landscape similarities enable warm-started optimization, reducing function evaluations by substantial margins. The work provides a scalable FLA framework and a rich dataset to guide future meta-heuristic design for software configuration tuning.

Abstract

Modern software systems are often highly configurable to tailor varied requirements from diverse stakeholders. Understanding the mapping between configurations and the desired performance attributes plays a fundamental role in advancing the controllability and tuning of the underlying system, yet has long been a dark hole of knowledge due to their black-box nature and the enormous combinatorial configuration space. In this paper, using $86$M evaluated configurations from three real-world systems on $32$ running workloads, we conducted one of its kind fitness landscape analysis (FLA) for configurable software systems. With comprehensive FLA methods, we for the first time show that: $i)$ the software configuration landscapes are fairly rugged, with numerous scattered local optima; $ii)$ nevertheless, the top local optima are highly accessible, featuring significantly larger basins of attraction; $iii)$ most inferior local optima are escapable with simple perturbations; $iv)$ landscapes of the same system with different workloads share structural similarities, which can be exploited to expedite heuristic search. Our results also provide valuable insights on the design of tailored meta-heuristics for configuration tuning; our FLA framework along with the collected data, build solid foundation for future research in this direction.

Figures (12)

• Figure 1: (A) Number of total configurations and local optima in each system, aggregated across workloads. (B) Autocorrelation calculated under different distances, aggregated across workloads. (C) Distribution of pairwise distance between local optima and $10^4$ randomly sampled configurations for LLVM-W1, where dashed lines are means. (D) Distribution of the distance of each local optimum to the global optimum for LLVM-W1. (E) Distribution of fitness values of all configurations local optima for LLVM-W1. (F) Average local optimum fitness (in percentile, lower is better) versus the corresponding distance to the global optimum for each system.
• Figure 2: Spearman correlation between local optima fitness and their properties for (A)LLVM, (B)SQLite, and (C)Apache.
• Figure 3: Example results on LLVM-W1: (A) boxplots and cumulative distribution of the size of best-improvement basins versus fitness percentile (the lower the better). (B) bars and cumulative distribution of frequency of visits for local optima at different fitness percentiles, during $10^9$ first-improvement local search runs. The long tails of both curves have been truncated. (C) distribution of basin size for local optima with fitness in the top, middle and bottom $0.1\%$ percentile based on first-improvement local search. (D) the distribution of accessibility to local optima basins of $10^3$ random configurations, where $1.0$ implies a configuration is shared by all basins. (E) overlap ratio between basins of the top $100$ local optima.
• Figure 4: Distribution of local optimum attributes calculated from LONs versus fitness percentile (the lower the better) for LLVM-W1.
• Figure 5: A part of the LLVM-W1 LON containing the global peaks. The global optimum and its nearest neighbors are colored in blue, and node radius indicates the basin size.