Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Breaking the $T^{2/3}$ Barrier for Sequential Calibration

Yuval Dagan, Constantinos Daskalakis, Maxwell Fishelson, Noah Golowich, Robert Kleinberg, Princewill Okoroafor

TL;DR

It is proved that the relationship between SPR and calibrated forecasting is bidirectional: not only do lower bounds for SPR translate into lower bounds for calibration, but algorithms for SPR also translate into new algorithms for calibrated forecasting.

Abstract

A set of probabilistic forecasts is calibrated if each prediction of the forecaster closely approximates the empirical distribution of outcomes on the subset of timesteps where that prediction was made. We study the fundamental problem of online calibrated forecasting of binary sequences, which was initially studied by Foster & Vohra (1998). They derived an algorithm with $O(T^{2/3})$ calibration error after $T$ time steps, and showed a lower bound of $Ω(T^{1/2})$. These bounds remained stagnant for two decades, until Qiao & Valiant (2021) improved the lower bound to $Ω(T^{0.528})$ by introducing a combinatorial game called sign preservation and showing that lower bounds for this game imply lower bounds for calibration. In this paper, we give the first improvement to the $O(T^{2/3})$ upper bound on calibration error of Foster & Vohra. We do this by introducing a variant of Qiao & Valiant's game that we call sign preservation with reuse (SPR). We prove that the relationship between SPR and calibrated forecasting is bidirectional: not only do lower bounds for SPR translate into lower bounds for calibration, but algorithms for SPR also translate into new algorithms for calibrated forecasting. We then give an improved \emph{upper bound} for the SPR game, which implies, via our equivalence, a forecasting algorithm with calibration error $O(T^{2/3 - \varepsilon})$ for some $\varepsilon > 0$, improving Foster & Vohra's upper bound for the first time. Using similar ideas, we then prove a slightly stronger lower bound than that of Qiao & Valiant, namely $Ω(T^{0.54389})$. Our lower bound is obtained by an oblivious adversary, marking the first $ω(T^{1/2})$ calibration lower bound for oblivious adversaries.

Breaking the $T^{2/3}$ Barrier for Sequential Calibration

TL;DR

It is proved that the relationship between SPR and calibrated forecasting is bidirectional: not only do lower bounds for SPR translate into lower bounds for calibration, but algorithms for SPR also translate into new algorithms for calibrated forecasting.

Abstract

A set of probabilistic forecasts is calibrated if each prediction of the forecaster closely approximates the empirical distribution of outcomes on the subset of timesteps where that prediction was made. We study the fundamental problem of online calibrated forecasting of binary sequences, which was initially studied by Foster & Vohra (1998). They derived an algorithm with calibration error after time steps, and showed a lower bound of . These bounds remained stagnant for two decades, until Qiao & Valiant (2021) improved the lower bound to by introducing a combinatorial game called sign preservation and showing that lower bounds for this game imply lower bounds for calibration. In this paper, we give the first improvement to the upper bound on calibration error of Foster & Vohra. We do this by introducing a variant of Qiao & Valiant's game that we call sign preservation with reuse (SPR). We prove that the relationship between SPR and calibrated forecasting is bidirectional: not only do lower bounds for SPR translate into lower bounds for calibration, but algorithms for SPR also translate into new algorithms for calibrated forecasting. We then give an improved \emph{upper bound} for the SPR game, which implies, via our equivalence, a forecasting algorithm with calibration error for some , improving Foster & Vohra's upper bound for the first time. Using similar ideas, we then prove a slightly stronger lower bound than that of Qiao & Valiant, namely . Our lower bound is obtained by an oblivious adversary, marking the first calibration lower bound for oblivious adversaries.
Paper Structure (65 sections, 41 theorems, 112 equations, 5 algorithms)

This paper contains 65 sections, 41 theorems, 112 equations, 5 algorithms.

Table of Contents

  1. Introduction
  2. Sequential calibration.
  3. Our contributions.
  4. Preliminaries
  5. Online Prediction and the $l_1$-calibration error
  6. Sign-Preservation Game with Reuse
  7. Overview of \ref{['thm:cal-ub']}: sign-preservation upper bound
  8. Case 1: imbalanced choices.
  9. Case 2: balanced choices.
  10. Case 2a: halves remain roughly balanced throughout.
  11. Case 2b: halves are very unbalanced at some point.
  12. Equivalence between Calibration and Sign Preservation
  13. Upper Bounding $\ell_1$-calibration error using Sign Preservation
  14. "Covering up" past errors.
  15. Biasing Calibration Error in a Specific Direction.
  16. ...and 50 more sections

Key Result

Theorem 1.2

If there exists $\varepsilon > 0$ such that for all $n \in \mathbb{N}$, $\mathrm{opt}(n,n) \leq O(n^{1-\varepsilon})$, then there exists a forecaster that guarantees calibration error of $O(T^{\frac{2}{3} - \frac{\varepsilon}{18}})$. If instead $\mathrm{opt}(n,n^\alpha) \geq \Omega(n^\beta)$ for som

Theorems & Definitions (82)

  • Theorem 1.2: Equivalence
  • Theorem 1.3: Upper bound for calibration
  • Theorem 1.4: Oblivious calibration adversary
  • Theorem 4.1
  • Theorem 4.2
  • Lemma 5.1
  • proof : Proof of \ref{['thm:cal-ub']}
  • Lemma 5.2
  • proof
  • Lemma 5.3
  • ...and 72 more