Adapting Multi-objectivized Software Configuration Tuning

TL;DR

AdMMO addresses local optima in software configuration tuning by adaptively adjusting the MMO weight $w$ and maintaining a controlled proportion $p$ of nondominated configurations through progressive triggering and partial duplicate retention. It combines an adaptive weight scheme with a retention mechanism to balance exploitation and exploration while preserving informative duplicates. In 14 system-objective pairs, AdMMO outperformed MMO and a broad set of baselines, delivering speedups from $2.2\times$ to $20\times$ and typically maintaining $p \approx 0.3$. These results demonstrate robust, budget-efficient tuning across diverse real-world software systems and budgets.

Abstract

When tuning software configuration for better performance (e.g., latency or throughput), an important issue that many optimizers face is the presence of local optimum traps, compounded by a highly rugged configuration landscape and expensive measurements. To mitigate these issues, a recent effort has shifted to focus on the level of optimization model (called meta multi-objectivization or MMO) instead of designing better optimizers as in traditional methods. This is done by using an auxiliary performance objective, together with the target performance objective, to help the search jump out of local optima. While effective, MMO needs a fixed weight to balance the two objectives-a parameter that has been found to be crucial as there is a large deviation of the performance between the best and the other settings. However, given the variety of configurable software systems, the "sweet spot" of the weight can vary dramatically in different cases and it is not possible to find the right setting without time-consuming trial and error. In this paper, we seek to overcome this significant shortcoming of MMO by proposing a weight adaptation method, dubbed AdMMO. Our key idea is to adaptively adjust the weight at the right time during tuning, such that a good proportion of the nondominated configurations can be maintained. Moreover, we design a partial duplicate retention mechanism to handle the issue of too many duplicate configurations without losing the rich information provided by the "good" duplicates. Experiments on several real-world systems, objectives, and budgets show that, for 71% of the cases, AdMMO is considerably superior to MMO and a wide range of state-of-the-art optimizers while achieving generally better efficiency with the best speedup between 2.2x and 20x.

Figures (10)

• Figure 2: The theoretical analysis and empirical evidence on the role of MMO weights for system MariaDB with normalized target and auxiliary performance objectives ($f_t$ is runtime and $f_a$ is CPU load). (a) shows the scaling on $f_a$ by a factor of $w=0.5$; (b) is the $45^{\circ}$ rotation and dilation on both axes by a factor of $\sqrt{2}$ afterwards. (c), (d), and (e) are the empirical results on configurations in an iteration with distinct $w$ thereof.
• Figure 3: The sensitivity of (improved) MMO to weight $w$ over 50 runs on different system-objective pairs. (a) shows the % improvement that can be made by the best $w$. (b) illustrates the best fixed $w$. (c) demonstrates the best $w$ throughout the tuning for system MariaDB using runtime as the target. The best $w$ (among $\{0.01,0.03,...,10\}$) is identified by Scott-Knott test scott1974cluster and the average performance results.
• Figure 4: The probability distribution in progressive trigger of AdMMO. The red dot denotes $o=5$: 5 consecutive iterations in which the best configuration has not been changed. Clearly, even with the same $o$, there is a higher probability of trigger in (b) than (a) as the former consumes more measurements.
• Figure 5: Correlation between the actual proportion of nondominated configurations $p'$ and the weight $w$ under the (improved) MMO in one run (population size of 10 and without duplicates). $f_t$ and $f_a$ of Storm are throughput and latency, receptively; for MongoDB, the $f_t$ is runtime while $f_a$ is CPU load.
• Figure 6: The results from different ways of handling duplicates when preserving configurations for the next iteration under the MMO meta-objectives space. $\boldsymbol{x_2}$, $\boldsymbol{x_3}$, and $\boldsymbol{x_4}$ are duplicate; $\boldsymbol{x_6}$ and $\boldsymbol{x_7}$ are also duplicate.