Confidence and discoveries with e-values

TL;DR

A procedure that is based on e-values is introduced and it is shown that it is efficient both computationally and statistically using simulated and real-world datasets.

Abstract

We discuss systematically two versions of confidence regions: those based on p-values and those based on e-values, a recent alternative to p-values. Both versions can be applied to multiple hypothesis testing, and in this paper we are interested in procedures that control the number of false discoveries under arbitrary dependence between the base p- or e-values. We introduce a procedure that is based on e-values and show that it is efficient both computationally and statistically using simulated and real-world datasets. Comparison with the corresponding standard procedure based on p-values is not straightforward, but there are indications that the new one performs significantly better in some situations.

Figures (13)

• Figure 1: Cournot's principle and its two generalizations
• Figure 2: The p-values (black solid lines) and e-values on the decimal log scale for testing the null hypothesis $N(0,1)$. Left panel: for the signed $\chi$ alternatives. Right panel: for the alternatives in the family $N(\mu,1)$.
• Figure 3: The arithmetic-mean discovery matrix for 10 false and 10 true null hypotheses, as described in text. The colour map on the right gives Jeffreys's thresholds, the boundaries between different colours in most of our plots, on the decimal log scale. Row 10 is highlighted in blue.
• Figure 4: Upper left panel: the arithmetic-mean discovery matrix for 100 false and 100 true null hypotheses. Upper middle panel: the GWGS discovery p-matrix in the same situation for Fisher's thresholds $1\%$ and $5\%$ (with values below $1\%$ shown in red and between $1\%$ and $5\%$ in yellow) under arbitrary dependence. Upper right panel: as the upper middle panel but assuming independence. Lower left panel: the e-to-p calibrated arithmetic-mean discovery matrix in the upper left panel using Fisher's thresholds. Lower middle panel: the VS-transformed GWGS discovery p-matrix in the upper middle panel of Figure \ref{['fig:big']} (under arbitrary dependence) using Jeffreys's thresholds. Lower right panel: the VS-transformed GWGS discovery p-matrix in the upper right panel (under independence) using Jeffreys's thresholds.
• Figure 5: Left panel: the top-left $200\times200$ corner of the arithmetic-mean discovery matrix for the BRCA dataset for $B:=10000$, using Jeffreys's thresholds. Right panel: its version (based on \ref{['eq:permutation_e_simple']}) that is only approximately valid.

Theorems & Definitions (12)

• Remark 5.1
• Remark 6.1
• Remark 6.2
• Proposition 6.3
• proof
• Proposition 6.4
• proof
• Remark 6.5
• Proposition B.1
• proof