Robust Bayesian Optimisation with Unbounded Corruptions

TL;DR

The work tackles Bayesian Optimization under extreme, potentially infinite outliers by modeling adversaries that are bounded in frequency rather than magnitude. It introduces Robust Conjugate Gaussian Processes (RCGPs) with plateau-weighted P-IMQ to maintain GP-like updates when data are uncontaminated and down-weight corrupted observations, enabling two algorithms, FC-RCGP-UCB and A2-RCGP-UCB, to achieve zero-cost robustness and sublinear regret under frequency-constrained corruption. The theoretical guarantees show regret scaling with the corruption budget and the kernel’s information gain, while experiments across synthetic and real tasks validate robustness without sacrificing performance in clean settings. The results offer a principled path to safe and reliable BO in adversarial or failure-prone environments, with practical guidance on parameters and adaptive centering. Overall, the paper contributes a provably robust BO framework extending GP-UCB to handle unbounded corruptions with meaningful guarantees and practical applicability.

Abstract

Bayesian Optimization is critically vulnerable to extreme outliers. Existing provably robust methods typically assume a bounded cumulative corruption budget, which makes them defenseless against even a single corruption of sufficient magnitude. To address this, we introduce a new adversary whose budget is only bounded in the frequency of corruptions, not in their magnitude. We then derive RCGP-UCB, an algorithm coupling the famous upper confidence bound (UCB) approach with a Robust Conjugate Gaussian Process (RCGP). We present stable and adaptive versions of RCGP-UCB, and prove that they achieve sublinear regret in the presence of up to $O(T^{1/2})$ and $O(T^{1/3})$ corruptions with possibly infinite magnitude. This robustness comes at near zero cost: without outliers, RCGP-UCB's regret bounds match those of the standard GP-UCB algorithm.

Figures (5)

• Figure 1: Cumulative regret for different BO algorithms on a corrupted Forrester example. Approaches based on standard GPs or Student-t processes perform worse than RCGP-based methods. DiagnosticsGP performs well on this example, but does not satisfy any regret bounds.
• Figure 2: Comparison of the IMQ and P-IMQ weight functions. We use $W=1$, $c=1$, $g(\boldsymbol{x})=0$, and $L=2$.
• Figure 3: Zero-cost robustness validation: Cumulative regrets of the different BO algorithms in the uncorrupted Forrester experiment ($T_{\text{c}}=0$).
• Figure 4: Cumulative regrets for different BO algorithms applied to hyperparameter optimization on CIFAR with simulated training crashes.
• Figure 5: Cumulative regret of linear policy optimization on the 36D Lunar-Lander-3 environment.

Theorems & Definitions (36)

• Definition 1: Frequency-Constrained Corruption
• Definition 2: P-IMQ Weight
• Theorem 1: Zero-Cost Robustness
• Theorem 2: Regret for FC-RCGP-UCB
• Theorem 3: Regret for A2-RCGP-UCB
• Lemma 1
• proof
• Lemma 2
• proof
• Lemma 3