Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning

TL;DR

This work addresses cost-sensitive hyperparameter optimization by integrating a utility-based objective into freeze-thaw Bayesian optimization. It introduces CFBO, which couples a user-defined utility U(b, \tilde{y}_b) with a nonmyopic EI-based acquisition and an adaptive stopping rule, while leveraging LC mixup transfer learning to improve LC extrapolation. The authors validate CFBO on standard multi-fidelity HPO benchmarks and a real-world object-detection dataset, showing improved cost-performance trade-offs and robust performance across diverse utility forms. The approach offers practical gains for users with budget constraints, enabling principled early stopping and resource-aware optimization in complex training regimes.

Abstract

In this paper, we address the problem of \emph{cost-sensitive} hyperparameter optimization (HPO) built upon freeze-thaw Bayesian optimization (BO). Specifically, we assume a scenario where users want to early-stop the HPO process when the expected performance improvement is not satisfactory with respect to the additional computational cost. Motivated by this scenario, we introduce \emph{utility} in the freeze-thaw framework, a function describing the trade-off between the cost and performance that can be estimated from the user's preference data. This utility function, combined with our novel acquisition function and stopping criterion, allows us to dynamically continue training the configuration that we expect to maximally improve the utility in the future, and also automatically stop the HPO process around the maximum utility. Further, we improve the sample efficiency of existing freeze-thaw methods with transfer learning to develop a specialized surrogate model for the cost-sensitive HPO problem. We validate our algorithm on established multi-fidelity HPO benchmarks and show that it outperforms all the previous freeze-thaw BO and transfer-BO baselines we consider, while achieving a significantly better trade-off between the cost and performance. Our code is publicly available at https://github.com/db-Lee/CFBO.

Figures (23)

• Figure 1: A concept of user utility.
• Figure 2: An example of utility function estimation.
• Figure 3: An illustration of $\boldsymbol{A}$in \ref{['eq:acquisition']}.
• Figure 4: (a)$\delta_b$ vs. $p_b$ for varying $\boldsymbol{\gamma}$ with $\beta=\exp(-1)$. (b)$\delta_b$ vs. $p_b$ for varying$\boldsymbol{\beta}$ with $\gamma=\log_{0.5} 0.2$.
• Figure 5: Results on conventional multi-fidelity HPO ($\alpha = 0$). X and O markers denote non-transfer and transfer learning methods, respectively.