On the Distance from Calibration in Sequential Prediction

TL;DR

This work introduces the calibration distance and its lower-calibration relaxation for sequential binary prediction, establishing that CalDist can be efficiently approximated up to an additive $O(\sqrt{T})$ and that there exists an adversarial forecaster achieving an $O(\sqrt{T})$ calibration distance in expectation. The core technical advance is relating CalDist to the lower calibration distance and proving a tight additive gap via rounding lemmas that preserve calibration while controlling support size. The authors also analyze performance under random-bit adversaries, showing a surprising polylogarithmic upper bound on CalDist in some regimes, and proving an $\Omega(T^{1/3})$ lower bound even with limited random generation and a potential early stopping. Collectively, the paper advances understanding of calibration in sequential prediction, connects transport-based interpretations to online learning frameworks, and presents a path toward explicit, efficiently computable calibration guarantees. The results have implications for designing calibration-aware forecasters in adversarial settings and for evaluating calibration-distance metrics in sequential decision-making contexts.

Abstract

We study a sequential binary prediction setting where the forecaster is evaluated in terms of the calibration distance, which is defined as the $L_1$ distance between the predicted values and the set of predictions that are perfectly calibrated in hindsight. This is analogous to a calibration measure recently proposed by Błasiok, Gopalan, Hu and Nakkiran (STOC 2023) for the offline setting. The calibration distance is a natural and intuitive measure of deviation from perfect calibration, and satisfies a Lipschitz continuity property which does not hold for many popular calibration measures, such as the $L_1$ calibration error and its variants. We prove that there is a forecasting algorithm that achieves an $O(\sqrt{T})$ calibration distance in expectation on an adversarially chosen sequence of $T$ binary outcomes. At the core of this upper bound is a structural result showing that the calibration distance is accurately approximated by the lower calibration distance, which is a continuous relaxation of the former. We then show that an $O(\sqrt{T})$ lower calibration distance can be achieved via a simple minimax argument and a reduction to online learning on a Lipschitz class. On the lower bound side, an $Ω(T^{1/3})$ calibration distance is shown to be unavoidable, even when the adversary outputs a sequence of independent random bits, and has an additional ability to early stop (i.e., to stop producing random bits and output the same bit in the remaining steps). Interestingly, without this early stopping, the forecaster can achieve a much smaller calibration distance of $\mathrm{polylog}(T)$.

Theorems & Definitions (53)

• Theorem 1
• Theorem 2
• Theorem 3
• Proposition 1
• Theorem 4
• Definition 1: Lower Calibration Distance
• Remark 2
• Lemma 3: Theorem 7.3 of BGHN23
• Proposition 4
• proof