Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Calibrated Probabilistic Forecasts for Arbitrary Sequences

Charles Marx, Volodymyr Kuleshov, Stefano Ermon

TL;DR

Calibrated Probabilistic Forecasts for Arbitrary Sequences introduces a universal online forecasting framework that guarantees calibrated uncertainty under nonstationary and adversarial data by leveraging Blackwell approachability. By expressing calibration notions as vector payoffs, the authors prove an $O(1/\sqrt{T})$ miscalibration bound and develop a unified recalibration recipe capable of handling multiple calibration objectives simultaneously. They instantiate this framework with ORCA, a gradient-based algorithm for online recalibration, and provide tractable specialized oracles for quantile, distribution, moment-based, and decision calibration, along with no-regret guarantees relative to expert forecasters. Empirical results on wind-energy and solar-physics datasets show improved calibration and downstream decision performance, highlighting practical impact for energy systems and other dynamic domains where robust uncertainty estimates are essential.

Abstract

Real-world data streams can change unpredictably due to distribution shifts, feedback loops and adversarial actors, which challenges the validity of forecasts. We present a forecasting framework ensuring valid uncertainty estimates regardless of how data evolves. Leveraging the concept of Blackwell approachability from game theory, we introduce a forecasting framework that guarantees calibrated uncertainties for outcomes in any compact space (e.g., classification or bounded regression). We extend this framework to recalibrate existing forecasters, guaranteeing calibration without sacrificing predictive performance. We implement both general-purpose gradient-based algorithms and algorithms optimized for popular special cases of our framework. Empirically, our algorithms improve calibration and downstream decision-making for energy systems.

Calibrated Probabilistic Forecasts for Arbitrary Sequences

TL;DR

Calibrated Probabilistic Forecasts for Arbitrary Sequences introduces a universal online forecasting framework that guarantees calibrated uncertainty under nonstationary and adversarial data by leveraging Blackwell approachability. By expressing calibration notions as vector payoffs, the authors prove an miscalibration bound and develop a unified recalibration recipe capable of handling multiple calibration objectives simultaneously. They instantiate this framework with ORCA, a gradient-based algorithm for online recalibration, and provide tractable specialized oracles for quantile, distribution, moment-based, and decision calibration, along with no-regret guarantees relative to expert forecasters. Empirical results on wind-energy and solar-physics datasets show improved calibration and downstream decision performance, highlighting practical impact for energy systems and other dynamic domains where robust uncertainty estimates are essential.

Abstract

Real-world data streams can change unpredictably due to distribution shifts, feedback loops and adversarial actors, which challenges the validity of forecasts. We present a forecasting framework ensuring valid uncertainty estimates regardless of how data evolves. Leveraging the concept of Blackwell approachability from game theory, we introduce a forecasting framework that guarantees calibrated uncertainties for outcomes in any compact space (e.g., classification or bounded regression). We extend this framework to recalibrate existing forecasters, guaranteeing calibration without sacrificing predictive performance. We implement both general-purpose gradient-based algorithms and algorithms optimized for popular special cases of our framework. Empirically, our algorithms improve calibration and downstream decision-making for energy systems.
Paper Structure (73 sections, 23 theorems, 41 equations, 2 figures, 1 table, 2 algorithms)

This paper contains 73 sections, 23 theorems, 41 equations, 2 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Setting
  3. Online Learning
  4. Calibration
  5. Blackwell Approachability
  6. The Forecasting Game
  7. Binary Calibration
  8. Quantile Calibration
  9. Moment Matching
  10. Decision Calibration
  11. A General Recipe
  12. Winning the Forecasting Game with Provable Calibration
  13. An Existence Result for Online Calibration
  14. Multi-Objective and No-Regret Recalibration
  15. Algorithms
  16. ...and 58 more sections

Key Result

Proposition 4.0

If the payoff is bounded (Condition cond:boundedness), then

Figures (2)

  • Figure 1: Comparison of two recalibration techniques, isotonic kuleshov2018accurate and ORCA (ours) on real-world data. The goal is to reduce miscalibration (QCE), while maintaining predictive performance (SMAPE). The best value is bolded and values within 10% of best are underlined.
  • Figure 2: Comparison of the decision loss incurred by decisions based on expert forecasts before and after recalibration with ORCA. The solid line indicates the instantaneous decision loss at each step, and the dashed line indicates the mean value over the entire period. Lower values are better.

Theorems & Definitions (37)

  • Definition 3.1
  • Proposition 4.0
  • Proposition 4.0
  • Theorem 4.1
  • Proposition 4.1
  • Theorem 5.1
  • Proposition A.0
  • proof
  • Proposition A.0
  • proof
  • ...and 27 more