Calibration of P-values for calibration and for deviation of a subpopulation from the full population

TL;DR

This work addresses calibrating P-values for two related problems: calibration of probabilistic predictions and deviation of a subpopulation from the full population under a conditioning covariate. It develops a Brownian-motion–based framework to calibrate KS- and Kuiper-type statistics, deriving computable CDFs for the range and the maximum absolute value of standard Brownian motion on $[0,1]$ and providing Corollaries that yield P-values in closed form as $1 - D(G/oldsymbol{\sigma})$ and $1 - F(H/oldsymbol{\sigma})$. The methodology handles ties via weighted samples, and offers two equivalent formulations that avoid random tie-breaking. The approach is validated numerically and demonstrated on real census data, with open-source software making the methods readily usable for calibration of probabilistic predictions and covariate-conditioned deviations in practice. The results enable reliable, efficient significance testing for calibration and subpopulation analysis in diverse applications.

Abstract

The author's recent research papers, "Cumulative deviation of a subpopulation from the full population" and "A graphical method of cumulative differences between two subpopulations" (both published in volume 8 of Springer's open-access "Journal of Big Data" during 2021), propose graphical methods and summary statistics, without extensively calibrating formal significance tests. The summary metrics and methods can measure the calibration of probabilistic predictions and can assess differences in responses between a subpopulation and the full population while controlling for a covariate or score via conditioning on it. These recently published papers construct significance tests based on the scalar summary statistics, but only sketch how to calibrate the attained significance levels (also known as "P-values") for the tests. The present article reviews and synthesizes work spanning many decades in order to detail how to calibrate the P-values. The present paper presents computationally efficient, easily implemented numerical methods for evaluating properly calibrated P-values, together with rigorous mathematical proofs guaranteeing their accuracy, and illustrates and validates the methods with open-source software and numerical examples.

Figures (8)

• Figure 1: Both plots graph $1 - F(x)$ versus $x$, where $F$ is defined in (\ref{['cdf']}) and is central to Corollary \ref{['corollary']}. The plot on the right uses a logarithmic scale for the vertical axis, unlike the plot on the left. The vertical dotted line indicates the value of $x$ corresponding to the mean of the distribution for which $F$ is the cumulative distribution function.
• Figure 2: Both plots graph $1 - D(x)$ versus $x$, where $D$ is defined in (\ref{['kscdf']}) and is central to Corollary \ref{['corollary']}. The plot on the right uses a logarithmic scale for the vertical axis, unlike the plot on the left. The vertical dotted line indicates the value of $x$ corresponding to the mean of the distribution for which $D$ is the cumulative distribution function.
• Figure 3: Both plots graph $1 - \Phi(x)$ versus $x$, where $\Phi$ is the cumulative distribution function for the standard normal distribution; $\Phi(x) = \int_{-\infty}^x \exp(-y^2/2) \, dy \, / \, \sqrt{2\pi}$. The plot on the right uses a logarithmic scale for the vertical axis, unlike the plot on the left.
• Figure 4: Calibration curves (empirical cumulative distribution functions under the null hypothesis of perfectly calibrated data) of the Kuiper P-value for calibration for sample sizes $n =$ 100, 1,000, 10,000; in each plot, the dashed line connects the origin $(0, 0)$ to the point $(1, 1)$ and illustrates perfect calibration, while the curve for $n =$ 10,000 is closest to perfect, $n =$ 1,000 is next closest, and $n =$ 100 is the farthest. Subfigure (a) uses scores equispaced on the unit interval $[0, 1]$, (b) squares each of the initially equispaced scores, and (c) takes the square root of each of the initially equispaced scores. The score $s$ is the predicted probability, with the expected response $r(s) = s$ to assess calibration. Each empirical cumulative distribution function plotted arises from 100,000 data sets generated independently while assuming the null hypothesis.
• Figure 5: Calibration curves (empirical cumulative distribution functions under the null hypothesis of perfectly calibrated data) of the Kolmogorov-Smirnov P-value for calibration for sample sizes $n =$ 100, 1,000, 10,000; in each plot, the dashed line connects the origin $(0, 0)$ to the point $(1, 1)$ and illustrates perfect calibration, while the curve for $n =$ 10,000 is closest to perfect, $n =$ 1,000 is next closest, and $n =$ 100 is the farthest. Subfigure (a) uses scores equispaced on the unit interval $[0, 1]$, (b) squares each of the initially equispaced scores, and (c) takes the square root of each of the initially equispaced scores. The score $s$ is the predicted probability, with the expected response $r(s) = s$ to assess calibration. Each empirical cumulative distribution function plotted arises from 100,000 data sets generated independently while assuming the null hypothesis.

Theorems & Definitions (11)

• Theorem 1
• proof
• Theorem 2
• Theorem 3
• Theorem 4
• proof
• Theorem 5
• Theorem 6
• Theorem 7
• Theorem 8