Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The power of forgetting in statistical hypothesis testing

Vladimir Vovk

Abstract

This paper places conformal testing in a general framework of statistical hypothesis testing. A standard approach to testing a composite null hypothesis $H$ is to test each of its elements and to reject $H$ when each of its elements is rejected. It turns out that we can fully cover conformal testing using this approach only if we allow forgetting some of the data. However, we will see that the standard approach covers conformal testing in a weak asymptotic sense and under restrictive assumptions. I will also list several possible directions of further research, including developing a general scheme of online testing.

The power of forgetting in statistical hypothesis testing

Abstract

This paper places conformal testing in a general framework of statistical hypothesis testing. A standard approach to testing a composite null hypothesis is to test each of its elements and to reject when each of its elements is rejected. It turns out that we can fully cover conformal testing using this approach only if we allow forgetting some of the data. However, we will see that the standard approach covers conformal testing in a weak asymptotic sense and under restrictive assumptions. I will also list several possible directions of further research, including developing a general scheme of online testing.
Paper Structure (17 sections, 2 theorems, 48 equations, 1 figure)

This paper contains 17 sections, 2 theorems, 48 equations, 1 figure.

Table of Contents

  1. Introduction
  2. From batch to online hypothesis testing
  3. Three modern ways of dynamic hypothesis testing
  4. Element-wise testing
  5. Pivotal testing
  6. Conformal testing
  7. General scheme of online testing
  8. Need for forgetting
  9. An example for pivotal testing
  10. An example for conformal testing
  11. Another way of forgetting
  12. Element-wise testing partially works for a fixed horizon
  13. Finite horizon
  14. Infinite horizon
  15. Creating natural test martingales out of likelihood ratios
  16. ...and 2 more sections

Key Result

Proposition 6.1

Let $N\in\{1,2,\dots\}$, and let $(S^\theta)$ be a family of test martingales w.r. to the same filtration (perhaps not natural) and a statistical model $(P_{\theta})$. Then there exists a family of natural test martingales $(\tilde{S}^\theta)$ such that

Figures (1)

  • Figure 1: Left panel: Five final values as described in text for a fixed dataset (for a changepoint detection problem). Right panel: Five final values as described in text for random datasets.

Theorems & Definitions (12)

  • Remark 2.1
  • Remark 2.2
  • Remark 3.1
  • Remark 3.2
  • Remark 3.3
  • Remark 3.4
  • Remark 3.5
  • Remark 3.6
  • Proposition 6.1
  • proof : Proof of Proposition \ref{['prop:exact']}
  • ...and 2 more