Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Truthfulness of Calibration Measures

Nika Haghtalab, Mingda Qiao, Kunhe Yang, Eric Zhao

Abstract

We initiate the study of the truthfulness of calibration measures in sequential prediction. A calibration measure is said to be truthful if the forecaster (approximately) minimizes the expected penalty by predicting the conditional expectation of the next outcome, given the prior distribution of outcomes. Truthfulness is an important property of calibration measures, ensuring that the forecaster is not incentivized to exploit the system with deliberate poor forecasts. This makes it an essential desideratum for calibration measures, alongside typical requirements, such as soundness and completeness. We conduct a taxonomy of existing calibration measures and their truthfulness. Perhaps surprisingly, we find that all of them are far from being truthful. That is, under existing calibration measures, there are simple distributions on which a polylogarithmic (or even zero) penalty is achievable, while truthful prediction leads to a polynomial penalty. Our main contribution is the introduction of a new calibration measure termed the Subsampled Smooth Calibration Error (SSCE) under which truthful prediction is optimal up to a constant multiplicative factor.

Truthfulness of Calibration Measures

Abstract

We initiate the study of the truthfulness of calibration measures in sequential prediction. A calibration measure is said to be truthful if the forecaster (approximately) minimizes the expected penalty by predicting the conditional expectation of the next outcome, given the prior distribution of outcomes. Truthfulness is an important property of calibration measures, ensuring that the forecaster is not incentivized to exploit the system with deliberate poor forecasts. This makes it an essential desideratum for calibration measures, alongside typical requirements, such as soundness and completeness. We conduct a taxonomy of existing calibration measures and their truthfulness. Perhaps surprisingly, we find that all of them are far from being truthful. That is, under existing calibration measures, there are simple distributions on which a polylogarithmic (or even zero) penalty is achievable, while truthful prediction leads to a polynomial penalty. Our main contribution is the introduction of a new calibration measure termed the Subsampled Smooth Calibration Error (SSCE) under which truthful prediction is optimal up to a constant multiplicative factor.
Paper Structure (72 sections, 34 theorems, 202 equations, 2 tables)

This paper contains 72 sections, 34 theorems, 202 equations, 2 tables.

Table of Contents

  1. Introduction
  2. Part I: We show that existing calibration measures do not simultaneously meet these criteria.
  3. Part II: We introduce a new calibration measure, called SSCE, that is sound, complete, and approximately truthful.
  4. Part III: There is a forecasting algorithm that achieves $O(\sqrt{T})$ SSCE even in the adversarial setting.
  5. Related Work
  6. Sequential calibration.
  7. Calibration measures.
  8. Subsampling.
  9. Proper scoring rules.
  10. Preliminaries
  11. Sequential prediction.
  12. Calibration measures.
  13. Completeness and soundness.
  14. Truthfulness.
  15. Technical Overview
  16. ...and 57 more sections

Key Result

Theorem 1.1

For every product distribution with marginals bounded away from $0$ and $1$, the truthful forecaster incurs $\Omega(\sqrt{T})$ smooth calibration error but there exists a forecasting algorithm that incurs only $\mathop{\mathrm{polylog}}\nolimits(T)$ smooth calibration error.

Theorems & Definitions (72)

  • Theorem 1.1: Informal
  • Theorem 1.2: Main Theorem
  • Theorem 1.3
  • Definition 2.1: Subsampled Smooth Calibration Error
  • Definition 2.2: Completeness and soundness
  • Definition 2.3: Truthful forecaster
  • Definition 2.4: Optimal error
  • Definition 2.5: Truthfulness of calibration measures
  • Theorem 5.1
  • proof : Proof sketch
  • ...and 62 more