Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Elementary Predictor Obtaining $2\sqrt{T}+1$ Distance to Calibration

Eshwar Ram Arunachaleswaran, Natalie Collina, Aaron Roth, Mirah Shi

TL;DR

This work tackles the problem of achieving small distance to calibration in adversarial online binary prediction by providing an explicit, efficient deterministic algorithm. Building on prior results that the minimax solution can reach $O(\sqrt{T})$ distance, the authors introduce Almost-One-Step-Ahead, which operates on a discretized prediction grid and emulates the lookahead behavior of a One-Step-Ahead benchmark. The main result proves CalDist $\le 2\sqrt{T}+1$ for any outcome sequence, achieved by setting the grid parameter to $m=\sqrt{T}$. This yields a practical, provably tight algorithm for distance to calibration, improving over prior non-constructive guarantees and offering a concrete tool for online calibration in adversarial settings.

Abstract

Blasiok et al. [2023] proposed distance to calibration as a natural measure of calibration error that unlike expected calibration error (ECE) is continuous. Recently, Qiao and Zheng [2024] gave a non-constructive argument establishing the existence of an online predictor that can obtain $O(\sqrt{T})$ distance to calibration in the adversarial setting, which is known to be impossible for ECE. They leave as an open problem finding an explicit, efficient algorithm. We resolve this problem and give an extremely simple, efficient, deterministic algorithm that obtains distance to calibration error at most $2\sqrt{T}+1$.

An Elementary Predictor Obtaining $2\sqrt{T}+1$ Distance to Calibration

TL;DR

This work tackles the problem of achieving small distance to calibration in adversarial online binary prediction by providing an explicit, efficient deterministic algorithm. Building on prior results that the minimax solution can reach distance, the authors introduce Almost-One-Step-Ahead, which operates on a discretized prediction grid and emulates the lookahead behavior of a One-Step-Ahead benchmark. The main result proves CalDist for any outcome sequence, achieved by setting the grid parameter to . This yields a practical, provably tight algorithm for distance to calibration, improving over prior non-constructive guarantees and offering a concrete tool for online calibration in adversarial settings.

Abstract

Blasiok et al. [2023] proposed distance to calibration as a natural measure of calibration error that unlike expected calibration error (ECE) is continuous. Recently, Qiao and Zheng [2024] gave a non-constructive argument establishing the existence of an online predictor that can obtain distance to calibration in the adversarial setting, which is known to be impossible for ECE. They leave as an open problem finding an explicit, efficient algorithm. We resolve this problem and give an extremely simple, efficient, deterministic algorithm that obtains distance to calibration error at most .
Paper Structure (3 sections, 3 theorems, 10 equations, 1 algorithm)

This paper contains 3 sections, 3 theorems, 10 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. Setting
  3. Analysis of Algorithm \ref{['alg:1']}

Key Result

Lemma 1

Fix a sequence of predictions $p^{1:T}$ and outcomes $y^{1:T}$. Then, $\mathrm{CalDist}(p^{1:T}, y^{1:T}) \leq \mathrm{ECE}(p^{1:T}, y^{1:T})$.

Theorems & Definitions (5)

  • Lemma 1: qiao2024distance
  • proof
  • Theorem 1
  • Theorem 2
  • proof