Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Calibration Bands for Mean Estimates within the Exponential Dispersion Family

Łukasz Delong, Selim Gatti, Mario V. Wüthrich

TL;DR

This work develops non-asymptotic calibration bands for mean estimates within the exponential dispersion family (EDF), enabling tests of calibration and auto-calibration that operate for any sample size. By extending stochastic-order and convolution techniques from binary and sub-Gaussian settings to the whole EDF, the authors derive data-dependent bands built from weighted partial sums that hold with probability at least $1-\alpha$. They provide explicit closed-form bands for common discrete and continuous EDF members (e.g., binomial, Poisson, negative binomial, gamma, normal) and extend the construction to regression with a ranking function, along with methods to handle band crossings via isotonic recalibration. The approach is then demonstrated through numerical experiments and real-data applications, showing practical detection of calibration violations and robustness to dispersion parameter estimation, while highlighting computational strategies such as binning and selective pairing. The results offer a versatile non-asymptotic tool for assessing and testing calibration in EDF-based models with broad applicability in statistics and actuarial science.

Abstract

A statistical model is said to be calibrated if the resulting mean estimates perfectly match the true means of the underlying responses. Aiming for calibration is often not achievable in practice as one has to deal with finite samples of noisy observations. A weaker notion of calibration is auto-calibration. An auto-calibrated model satisfies that the expected value of the responses for a given mean estimate matches this estimate. Testing for autocalibration has only been considered recently in the literature and we propose a new approach based on calibration bands. Calibration bands denote a set of lower and upper bounds such that the probability that the true means lie simultaneously inside those bounds exceeds some given confidence level. Such bands were constructed by Yang-Barber (2019) for sub-Gaussian distributions. Dimitriadis et al. (2023) then introduced narrower bands for the Bernoulli distribution. We use the same idea in order to extend the construction to the entire exponential dispersion family that contains for example the binomial, Poisson, negative binomial, gamma and normal distributions. Moreover, we show that the obtained calibration bands allow us to construct various tests for calibration and auto-calibration, respectively. As the construction of the bands does not rely on asymptotic results, we emphasize that our tests can be used for any sample size.

Calibration Bands for Mean Estimates within the Exponential Dispersion Family

TL;DR

This work develops non-asymptotic calibration bands for mean estimates within the exponential dispersion family (EDF), enabling tests of calibration and auto-calibration that operate for any sample size. By extending stochastic-order and convolution techniques from binary and sub-Gaussian settings to the whole EDF, the authors derive data-dependent bands built from weighted partial sums that hold with probability at least . They provide explicit closed-form bands for common discrete and continuous EDF members (e.g., binomial, Poisson, negative binomial, gamma, normal) and extend the construction to regression with a ranking function, along with methods to handle band crossings via isotonic recalibration. The approach is then demonstrated through numerical experiments and real-data applications, showing practical detection of calibration violations and robustness to dispersion parameter estimation, while highlighting computational strategies such as binning and selective pairing. The results offer a versatile non-asymptotic tool for assessing and testing calibration in EDF-based models with broad applicability in statistics and actuarial science.

Abstract

A statistical model is said to be calibrated if the resulting mean estimates perfectly match the true means of the underlying responses. Aiming for calibration is often not achievable in practice as one has to deal with finite samples of noisy observations. A weaker notion of calibration is auto-calibration. An auto-calibrated model satisfies that the expected value of the responses for a given mean estimate matches this estimate. Testing for autocalibration has only been considered recently in the literature and we propose a new approach based on calibration bands. Calibration bands denote a set of lower and upper bounds such that the probability that the true means lie simultaneously inside those bounds exceeds some given confidence level. Such bands were constructed by Yang-Barber (2019) for sub-Gaussian distributions. Dimitriadis et al. (2023) then introduced narrower bands for the Bernoulli distribution. We use the same idea in order to extend the construction to the entire exponential dispersion family that contains for example the binomial, Poisson, negative binomial, gamma and normal distributions. Moreover, we show that the obtained calibration bands allow us to construct various tests for calibration and auto-calibration, respectively. As the construction of the bands does not rely on asymptotic results, we emphasize that our tests can be used for any sample size.

Paper Structure

This paper contains 29 sections, 9 theorems, 148 equations, 12 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Calibration bands within the exponential dispersion family
  3. Stochastic orders within the exponential dispersion family
  4. Construction of the calibration bands
  5. Main result
  6. Choice of the set of ordered pairs and binning of the observations
  7. Crossings inside the calibration bands
  8. Explicit calibration bands for selected distributions
  9. Continuous distributions
  10. Comparison with Yang--Barber's calibration bands
  11. Extension to regression modeling
  12. Regression modeling within the exponential dispersion family
  13. Construction of the calibration bands
  14. Statistical testing of calibration and auto-calibration
  15. The auto-calibration property
  16. ...and 14 more sections

Key Result

Proposition 3.4

Suppose that Assumptions assumption EDF and ass effective domain hold and let $\mu_1 \leq \mu_2$ be in the mean parameter space $\kappa'(\mathring{\Theta})$. Then, for any volume $v > 0$, dispersion parameter $\varphi > 0$ and cumulant function $\kappa$, the random variables $Y_1 \sim \textnormal{ED

Figures (12)

  • Figure 1: Calibration plot of the isotonic mean estimates of independent normal responses $(Y_i)_{i=1}^{n}$. The calibration band is plotted in red, whereas the mean estimates are drawn in black.
  • Figure 2: Calibration bands on the mean of independent normal responses $(Y_i)_{i=1}^{n}$ for various confidence levels.
  • Figure 3: Calibration bands on the mean of independent normal responses $(Y_i)_{i=1}^{n}$ that are constructed using the set $\mathcal{J}^{nbh}_{s}$ for various sizes $s$.
  • Figure 4: Calibration bands on the mean of independent normal responses $(Y_i)_{i=1}^{n}$ that are constructed using the set $\mathcal{J}^{dist}_{d}$ for various distances $d$.
  • Figure 5: Calibration bands on the mean constructed by binning independent normal responses $(Y_i)_{i=1}^{n}$ for different amounts of bins.
  • ...and 7 more figures

Theorems & Definitions (12)

  • Definition 3.3
  • Proposition 3.4
  • Proposition 3.5
  • Lemma 3.6
  • Proposition 3.7
  • Theorem 4.1
  • Proposition 5.1
  • Proposition 5.2
  • Remark 5.3
  • Theorem 5.4
  • ...and 2 more