Cost-aware Stopping for Bayesian Optimization

TL;DR

The paper addresses when to stop expensive Bayesian optimization in cost-varying environments by introducing a principled stopping rule linked to Pandora's Box/Gittins index and the LogEIPC EI-based criterion. The authors prove a theoretical bound on the cost-adjusted simple regret when the stopping rule is paired with either the PBGI or LogEIPC acquisition function, and they show the rule is equivalent to an EI-cost condition, enabling a unified framework. Empirically, the method competes with or outperforms baselines across synthetic Bayesian regret tasks and real-world hyperparameter and neural-architecture searches, especially under high evaluation costs, while offering practical guidance for unknown costs and budget constraints. The work delivers a tuning-free, theoretically grounded approach to cost-aware stopping in Bayesian optimization with broad applicability and strong practical impact.

Abstract

In automated machine learning, scientific discovery, and other applications of Bayesian optimization, deciding when to stop evaluating expensive black-box functions in a cost-aware manner is an important but underexplored practical consideration. A natural performance metric for this purpose is the cost-adjusted simple regret, which captures the trade-off between solution quality and cumulative evaluation cost. While several heuristic or adaptive stopping rules have been proposed, they lack guarantees ensuring stopping before incurring excessive function evaluation costs. We propose a principled cost-aware stopping rule for Bayesian optimization that adapts to varying evaluation costs without heuristic tuning. Our rule is grounded in a theoretical connection to state-of-the-art cost-aware acquisition functions, namely the Pandora's Box Gittins Index (PBGI) and log expected improvement per cost (LogEIPC). We prove a theoretical guarantee bounding the expected cost-adjusted simple regret incurred by our stopping rule when paired with either acquisition function. Across synthetic and empirical tasks, including hyperparameter optimization and neural architecture size search, pairing our stopping rule with PBGI or LogEIPC usually matches or outperforms other acquisition-function--stopping-rule pairs in terms of cost-adjusted simple regret.

Figures (27)

• Figure 1: Illustration of the PBGI/LogEIPC stopping rule under a uniform-cost setting. When the cost-per-sample is large ($c(x) \equiv 0.1$), the maximum LogEIPC acquisition value falls below the threshold $0.0$ and the minimum PBGI acquisition value exceeds the current best observed value, indicating stopping; when the cost-per-sample is small ($c(x)\equiv 0.0001$), the maximum LogEIPC acquisition value remains above the threshold $0.0$ and the minimum PBGI acquisition value is smaller than the current best observed value, indicating no stopping.
• Figure 2: Cost-adjusted simple regret across acquisition-stopping rule pairs in 1D and 8D Bayesian regret setting. In 1D, objective functions are sampled from a GP with a Matérn-5/2 kernel and a linear cost function scaled by $\lambda = 0.1, 0.01, 0.001$. The Immediate baseline is omitted at $\lambda = 0.001$ due to its much higher regret (mean 0.6942, error bar [0.5314, 0.8570]). In 8D, objective functions are also drawn from a GP with a Matérn-5/2 kernel, using three cost functions scaled by $\lambda = 0.01$.
• Figure 3: Cost-adjusted simple regret across acquisition function--stopping rule pairs on LCBench and NATS-Bench. The objective is to minimize validation error on classification tasks, with scaled proxy runtime as evaluation cost, scaled by representative values of $\lambda$ ($10^{-3}, 10^{-4}, 10^{-5}$ for LCBench and $10^{-5}$ for NATS-Bench). For LCBench, results are aggregated across 35 datasets using min–max normalization. Our PBGI/LogEIPC stopping rule, when paired with either LogEIPC or PBGI, typically ranks among the top 3 pairs and closely approaches the hindsight optimal on LCBench and on cifar10-valid in NATS-Bench, but slightly worse on the other two NATS datasets.
• Figure 4: Comparison of the raw and moving-averaged PBGI/LogEIPC stopping rule signals (i.e., the LogEIPC acquisition values) in Bayesian regret 8D experiments for multiple acquisition functions, with linear cost and stopping thresholds at $\log(0.1)$, $\log(0.01)$ and $\log(0.001)$. Left: The unaveraged signals exhibits large, high-frequency fluctuations due to the difficulty of acquisition optimization in high dimensions. Right: Applying moving average (window=20) smooths these wiggles, yielding more stable stopping signals.
• Figure 5: Surface plots of cost functions over $[0, 1]^2$: (Left) Uniform cost of $1$ across domain. (Middle) Normalized linear cost increasing with the mean of $x_1$ and $x_2$. (Right) Periodic cost with $\alpha=2$, $\beta=2$, normalized by Bessel-based factor.

Theorems & Definitions (15)

• Lemma 0
• Theorem 1
• Corollary 1
• Corollary 1
• Lemma 1
• proof
• Theorem 1
• proof
• Remark 2
• Remark 3