Robust and Computation-Aware Gaussian Processes

TL;DR

Gaussian Processes often struggle with scalability and robustness in large, outlier-contaminated data. The authors propose Robust Computation-Aware Gaussian Processes ($RCaGP$), which unifies robust-conjugate GP inference with computation-aware approximations, yielding conservative uncertainty and improved reliability. Theoretical results show robustness to outliers through a bounded posterior influence function and worst-case error guarantees, while an end-to-end optimization via $EULBO$ enables joint model and acquisition design. Empirically, $RCaGP$ outperforms baselines across regression and high-throughput BO tasks, and an expert-guided robust mean prior further enhances performance, signaling a practical impact for reliable large-scale probabilistic inference and optimization.

Abstract

Gaussian processes (GPs) are widely used for regression and optimization tasks such as Bayesian optimization (BO) due to their expressiveness and principled uncertainty estimates. However, in settings with large datasets corrupted by outliers, standard GPs and their sparse approximations struggle with computational tractability and robustness. We introduce Robust Computation-aware Gaussian Process (RCaGP), a novel GP model that jointly addresses these challenges by combining a principled treatment of approximation-induced uncertainty with robust generalized Bayesian updating. The key insight is that robustness and approximation-awareness are not orthogonal but intertwined: approximations can exacerbate the impact of outliers, and mitigating one without the other is insufficient. Unlike previous work that focuses narrowly on either robustness or approximation quality, RCaGP combines both in a principled and scalable framework, thus effectively managing both outliers and computational uncertainties introduced by approximations such as low-rank matrix multiplications. Our model ensures more conservative and reliable uncertainty estimates, a property we rigorously demonstrate. Additionally, we establish a robustness property and show that the mean function is key to preserving it, motivating a tailored model selection scheme for robust mean functions. Empirical results confirm that solving these challenges jointly leads to superior performance across both clean and outlier-contaminated settings, both on regression and high-throughput Bayesian optimization benchmarks.

Figures (8)

• Figure 1: Overview of our proposed RCaGP against concurrent baselines on a 1D example.(Left) SVGP fails to fit observed data contaminated by outliers, whereas RCaGP successfully fits high-density data regions while preserving a higher variance due to the presence of outliers. (Middle-left) While enhancing robustness compared to SVGP, RCSVGP deviates more significantly from the true function than RCaGP. (Middle-right) Even if CaGP displays increased posterior variance near outliers, RCaGP provides superior posterior mean prediction. (Right) As a result, the acquisition function landscape for RCaGP better prioritizes the true global optima.
• Figure 2: Expert-guided robust mean prior.(Left) Graphical model of expert's feedback generative process for identified outliers. (Middle) User graphical model for expert outlier corrections. (Right) Toy example: with expert corrections, the expert-guided prior better captures the true function than a constant mean prior under contamination.
• Figure 3: Left: Predictive mean of various approximate GP models on the DJIA index with a constant mean prior. Outliers affect the baselines, whereas RCaGP maintains robustness. Right: learned weight functions $w({\bf{x}},y)$ for RCaGP and RCSVGP, illustrating their influence on the regression.
• Figure 4: High-throughput Bayesian Optimization task. Each panel shows the best value found each iteration found so far, averaged across 20 repetitions $\pm$ 1 std. Columns 1-4 report results under asymmetric outliers contamination, with the $1^{\text{st}}$ row using the EULBO-EI acquisition function, the $2^{\text{nd}}$ row using DPPBO-EI. Column 5 features results without outliers using EULBO-EI.
• Figure 5: Ablation studies.(Left) Varying $c$ in weight function $w$ (Equation \ref{['eq:weight']}) on UCI regression datasets. MAE and NLL have been normalized for each dataset; the metrics displayed represent the average. (Right) BO results for RCaGP and RCSVGP using expert-driven prior mean function $m$ in $w$. Results are averaged over across 20 repetitions, with $\pm$ 1 standard deviation being shown.

Theorems & Definitions (11)

• Proposition 4.0
• Proposition 4.0
• Proposition C.0
• Lemma D.1
• Proposition E.0
• Corollary E.0
• Lemma F.1
• Proposition F.2
• Proposition F.2
• Proposition F.2