Catoni-style confidence sequences for heavy-tailed mean estimation

TL;DR

The best among the three approaches -- the Catoni-style confidence sequence -- performs remarkably well in practice, essentially matching the state-of-the-art methods for $\sigma^2$-subGaussian data, and provably attains the $\sqrt{\log \log t/t}$ lower bound due to the law of the iterated logarithm.

Abstract

A confidence sequence (CS) is a sequence of confidence intervals that is valid at arbitrary data-dependent stopping times. These are useful in applications like A/B testing, multi-armed bandits, off-policy evaluation, election auditing, etc. We present three approaches to constructing a confidence sequence for the population mean, under the minimal assumption that only an upper bound $σ^2$ on the variance is known. While previous works rely on light-tail assumptions like boundedness or subGaussianity (under which all moments of a distribution exist), the confidence sequences in our work are able to handle data from a wide range of heavy-tailed distributions. The best among our three methods -- the Catoni-style confidence sequence -- performs remarkably well in practice, essentially matching the state-of-the-art methods for $σ^2$-subGaussian data, and provably attains the $\sqrt{\log \log t/t}$ lower bound due to the law of the iterated logarithm. Our findings have important implications for sequential experimentation with unbounded observations, since the $σ^2$-bounded-variance assumption is more realistic and easier to verify than $σ^2$-subGaussianity (which implies the former). We also extend our methods to data with infinite variance, but having $p$-th central moment ($1<p<2$).

Figures (8)

• Figure 1: Self-normalized confidence sequence $\{ \operatorname{CI}^{\mathsf{SN}}_t \}$ under $X_t \overset{\text{i.i.d.}}{\sim} \mathcal{N}({0,1})$, $\lambda_t = 1/\sqrt{t}$, and $\alpha = 0.05$. The trichotomous topology \ref{['eqn:cisn-decomp']} manifests after around $t = 30$.
• Figure 2: Catoni-style confidence sequence $\{ \operatorname{CI}^{\mathsf{C}}_t \}$ under $X_t \overset{\text{i.i.d.}}{\sim} \mathcal{N}({0,1})$, $\lambda_t = \min\{1/\sqrt{t} , 0.1 \}$, and $\alpha = 0.05$. Notice the difference in scale of the $y$-axis from Figure \ref{['fig:sncs']}.
• Figure 3: To achieve the same level of lower tightness (e.g. when lower confidence bound surpasses 0), the trivial Catoni CS needs a sample of size 880, about 4 times the Catoni-style CS which only takes 246.
• Figure 4: Cumulative miscoverage rates when continuously monitoring CSs and CI under $\mathrm{t}_3$ distribution, which (provably) grow without bound for the Catoni CI, but are guaranteed to stay within $\alpha = 0.05$ for CSs.
• Figure 5: Comparison of CI/CS growth rates at $t =250$. In both figures, triangular markers denote random widths (for which we repeat 10 times to let randomness manifest), and square markers deterministic widths; hollow markers denote the widths of CIs, while filled markers the widths of CSs. Our Catoni-style CS is among the best CSs (even CIs) in terms of tightness under small error probability $\alpha$, in heavy and light tail regimes alike. In the right figure, note the overlap of the Chernoff-CI with Catoni-CI, as well as that of the Hoeffding-type subGaussian CS with the Catoni-style CS.

Theorems & Definitions (43)

• Lemma 1: Dubins-Savage inequality
• Theorem 2: Dubins-Savage confidence sequence
• Lemma 3
• Lemma 4: Ville's inequality
• Lemma 5
• Lemma 6: Self-normalized anticonfidence sequence
• Theorem 7: Self-normalized confidence sequence
• Lemma 8: Catoni supermartingales
• Theorem 9: Catoni-style confidence sequence
• Theorem 10