Practical, Utilitarian Algorithm Configuration

TL;DR

The paper strengthens a theoretically grounded utilitarian approach to algorithm configuration by delivering four practical improvements to COUP: KL-based confidence bounds, LUCB-based best-arm identification, adaptive configuration expansion, and model-guided search with gradient-boosted trees. Experimental results on ACLib demonstrate that the enhanced COUP matches the empirical performance of SMAC while maintaining formal guarantees, with notable gains in tightness of optimality bounds and speed of finding strong configurations. A SAT Competition case study illustrates how runtime-CDF analyses can reveal robustness of solver rankings to variations in the utility function. Overall, the work argues that utilitarian algorithm configuration with guarantees is now practical for real-world use and can adapt to user preferences over runtime.

Abstract

Utilitarian algorithm configuration identifies a parameter setting for a given algorithm that maximizes a user's utility. Utility functions offer a theoretically well-grounded approach to optimizing decision-making under uncertainty and are flexible enough to capture a user's preferences over algorithm runtimes (e.g., they can describe a sharp cutoff after which a solution is no longer required, a per-hour cost for compute, or diminishing returns from algorithms that take longer to run). COUP is a recently-introduced utilitarian algorithm configuration procedure which was designed mainly to offer strong theoretical guarantees about the quality of the configuration it returns, with less attention paid to its practical performance. This paper closes that gap, bringing theoretically-grounded, utilitarian algorithm configuration to the point where it is competitive with widely used, heuristic configuration procedures that offer no performance guarantees. We present a series of improvements to COUP that improve its empirical performance without degrading its theoretical guarantees and demonstrate their benefit experimentally. Using a case study, we also illustrate ways of exploring the robustness of a given solution to the algorithm selection problem to variations in the utility function.

Figures (19)

• Figure 1: Illustration of the decomposition that leads to our adaptive strategy for incorporating new configurations.
• Figure 2: Left: Suboptimality guarantee provided by COUP, demonstrating the improvements achieved by using the KL bounds and the LUCB algorithm. Center: Lower confidence bounds for configurations sampled by COUP. Configurations sampled according to the model tend to be provably better and to improve with time as more training data is gathered. Right: The CDF of the difference in normalized final utility between COUP and SMAC. Most of the time, COUP and SMAC perform equivalently.
• Figure 3: Unlike SMAC, COUP also offers theoretical guarantees. In general, both $\epsilon$ and $\gamma$ are driven toward 0.
• Figure 4: The winning SAT solver is FOSD over the 16th ranked "PReLearn-kissat-PReLearn-tern-kissat.sh".
• Figure 5: The winning SAT solver is FOSD over the 21st ranked "Kissat_MAB_Rephases" for runtimes $> \;\sim 10$s.