Robust Predictive Uncertainty and Double Descent in Contaminated Bayesian Random Features
Michele Caprio, Katerina Papagiannouli, Siu Lun Chau, Sayan Mukherjee
TL;DR
This paper develops a robust Bayesian framework for Random Features regression by incorporating $\epsilon$-contaminated priors and $\eta$-contaminated likelihoods through credal sets and pessimistic generalized Bayes updating. It derives explicit lower/upper posterior predictive density bounds, introduces the Imprecise Highest Density Region (IHDR) for robust uncertainty quantification, and provides tractable variance bounds that retain the leading proportional-growth asymptotics of RFs. The authors show that predictive uncertainty forms affine envelopes around the classical Gaussian predictive and that the characteristic double-descent behavior of RF variance is preserved under bounded misspecification, with the envelope magnitude controlled by the contamination parameters. The IHDR admits an efficient outer approximation via adjusted Gaussian intervals, ensuring practical, computation-friendly robust predictive uncertainty in high-dimensional RF settings. Overall, the work establishes a robustness theory for Bayesian RF that delivers worst-case guarantees under misspecification while preserving essential predictive structure.
Abstract
We propose a robust Bayesian formulation of random feature (RF) regression that accounts explicitly for prior and likelihood misspecification via Huber-style contamination sets. Starting from the classical equivalence between ridge-regularized RF training and Bayesian inference with Gaussian priors and likelihoods, we replace the single prior and likelihood with $ε$- and $η$-contaminated credal sets, respectively, and perform inference using pessimistic generalized Bayesian updating. We derive explicit and tractable bounds for the resulting lower and upper posterior predictive densities. These bounds show that, when contamination is moderate, prior and likelihood ambiguity effectively acts as a direct contamination of the posterior predictive distribution, yielding uncertainty envelopes around the classical Gaussian predictive. We introduce an Imprecise Highest Density Region (IHDR) for robust predictive uncertainty quantification and show that it admits an efficient outer approximation via an adjusted Gaussian credible interval. We further obtain predictive variance bounds (under a mild truncation approximation for the upper bound) and prove that they preserve the leading-order proportional-growth asymptotics known for RF models. Together, these results establish a robustness theory for Bayesian random features: predictive uncertainty remains computationally tractable, inherits the classical double-descent phase structure, and is improved by explicit worst-case guarantees under bounded prior and likelihood misspecification.