Calibrated Regression Against An Adversary Without Regret
Shachi Deshpande, Charles Marx, Volodymyr Kuleshov
TL;DR
This work tackles probabilistic forecasting in online, non-stationary settings by introducing online calibrated regression. It provides a post-hoc recalibration algorithm that converts an uncalibrated baseline forecast into calibrated CDF predictions while guaranteeing vanishing regret relative to the baseline under proper losses, notably the CRPS. The framework supports adversarial data streams and yields calibrated uncertainties, with demonstrated improvements in Bayesian optimization and multiple regression benchmarks, including UCI datasets. The results offer a principled approach to uncertainty estimation and decision-making in sequential, non-iid environments, with theoretical guarantees and practical impact on rapid convergence and reliable risk assessment.
Abstract
We are interested in probabilistic prediction in online settings in which data does not follow a probability distribution. Our work seeks to achieve two goals: (1) producing valid probabilities that accurately reflect model confidence; and (2) ensuring that traditional notions of performance (e.g., high accuracy) still hold. We introduce online algorithms guaranteed to achieve these goals on arbitrary streams of data points, including data chosen by an adversary. Specifically, our algorithms produce forecasts that are (1) calibrated -- i.e., an 80% confidence interval contains the true outcome 80% of the time -- and (2) have low regret relative to a user-specified baseline model. We implement a post-hoc recalibration strategy that provably achieves these goals in regression; previous algorithms applied to classification or achieved (1) but not (2). In the context of Bayesian optimization, an online model-based decision-making task in which the data distribution shifts over time, our method yields accelerated convergence to improved optima.