Online Calibrated and Conformal Prediction Improves Bayesian Optimization

TL;DR

This paper addresses the problem that Bayesian optimization relies on predictive uncertainty that can be miscalibrated, especially under non-stationary conditions. It introduces online recalibration algorithms based on quantile pinball loss and online optimization to enforce calibration of forecasts, which can be plugged into any BO loop with minimal overhead. Theoretical results connect calibration to improved decision-making, providing bounds on calibration error and demonstrating that recalibration yields reliable acquisition function values. Empirically, calibrated Bayesian optimization achieves faster convergence and better optima on standard benchmarks and hyperparameter optimization tasks, demonstrating practical impact across domains where expensive evaluations are common.

Abstract

Accurate uncertainty estimates are important in sequential model-based decision-making tasks such as Bayesian optimization. However, these estimates can be imperfect if the data violates assumptions made by the model (e.g., Gaussianity). This paper studies which uncertainties are needed in model-based decision-making and in Bayesian optimization, and argues that uncertainties can benefit from calibration -- i.e., an 80% predictive interval should contain the true outcome 80% of the time. Maintaining calibration, however, can be challenging when the data is non-stationary and depends on our actions. We propose using simple algorithms based on online learning to provably maintain calibration on non-i.i.d. data, and we show how to integrate these algorithms in Bayesian optimization with minimal overhead. Empirically, we find that calibrated Bayesian optimization converges to better optima in fewer steps, and we demonstrate improved performance on standard benchmark functions and hyperparameter optimization tasks.

Figures (8)

• Figure 1: Comparison of Uncalibrated and Calibrated Bayesian Optimization on the Forrester Function (Green) Using the UCB Acquisition Function (Blue).
• Figure 2: Comparison of Bayesian Optimization in Benchmark Functions. In the top plots, we see that calibrated method reduces calibration error. The bottom plots show that on an average, the calibrated method identifies the minimum using less iterations.
• Figure 3: Comparison of Calibrated and Uncalibrated Bayesian Optimization on Six-hump-camel Function (2D) for Various Acquisition Functions. The top plots show that the calibrated method reduces calibration error. In the bottom plots, we see that the calibrated method identifies the minimum using fewer iterations.
• Figure 4: Comparison of Calibrated Bayesian Optimization with Additional Baselines. The calibrated method outperforms the modern Bayesian optimization baseline stanton2023bayesian and Adaptive UCB for Bayesian optimization with unknown kernel hyperparameters berkenkamp2019noregret. The acquisition function used here is UCB.
• Figure 5: Hyperparameter Optimization Experiments. Top: The average improvement made by the calibrated method over the uncalibrated method. Bottom: The best minimum found by each method per iteration.

Theorems & Definitions (15)

• Theorem 1
• Lemma 1
• Theorem 2
• proof : Proof (Sketch)
• Theorem 3
• Theorem 3
• proof
• Lemma 2
• proof
• Lemma 3