Active Sequential Two-Sample Testing

TL;DR

This work tackles two-sample testing when feature access is cheap but labeling is costly. It introduces an active sequential framework that adaptively queries labels for features predicted to have high dependency with the labels, constructing a likelihood-ratio statistic $w_n$ that uses a maximized prior and a classifier $Q_n(z|oldsymbol{s})$. The main contributions include an anytime-valid $p$-value guarantee via Ville’s inequality, an information-theoretic view linking active labeling to mutual information gains, and finite-sample as well as asymptotic analyses. The authors instantiate the framework with a bimodal query (BQ-AST) and demonstrate through synthetic data, MNIST, and an Alzheimer's dataset that active labeling yields higher testing power with substantially fewer labeled samples while controlling Type I error. The results indicate significant practical impact for label-efficient hypothesis testing in domains where expensive biomarkers or labels dominate testing costs.

Abstract

A two-sample hypothesis test is a statistical procedure used to determine whether the distributions generating two samples are identical. We consider the two-sample testing problem in a new scenario where the sample measurements (or sample features) are inexpensive to access, but their group memberships (or labels) are costly. To address the problem, we devise the first \emph{active sequential two-sample testing framework} that not only sequentially but also \emph{actively queries}. Our test statistic is a likelihood ratio where one likelihood is found by maximization over all class priors, and the other is provided by a probabilistic classification model. The classification model is adaptively updated and used to predict where the (unlabelled) features have a high dependency on labels; labeling the ``high-dependency'' features leads to the increased power of the proposed testing framework. In theory, we provide the proof that our framework produces an \emph{anytime-valid} $p$-value. In addition, we characterize the proposed framework's gain in testing power by analyzing the mutual information between the feature and label variables in asymptotic and finite-sample scenarios. In practice, we introduce an instantiation of our framework and evaluate it using several experiments; the experiments on the synthetic, MNIST, and application-specific datasets demonstrate that the testing power of the instantiated active sequential test significantly increases while the Type I error is under control.

Figures (3)

• Figure 1: The active sequential two-sample testing framework.
• Figure 2: Under $H_0$ in which $\delta=0$, empirical Type I errors of the proposed test for different $P(Z=0)$ when using the logistic regression to build $Q(z\mid \mathbf{s})$. All Type I errors are smaller than $\alpha=0.05$, which agrees with Theorem \ref{['ActiveTypeI']}.
• Figure 3: Empirical Type I errors of the proposed test for different $P(Z=0)$ in the MNIST experiment. SVM is used to build $Q(z\mid \mathbf{s})$. All Type I errors are smaller than $\alpha=0.05$, which agrees with Theorem \ref{['ActiveTypeI']}.

Theorems & Definitions (13)

• Theorem 5.1
• Definition 5.2
• Remark 5.3
• Theorem 5.4
• Remark 5.5
• Definition 5.6
• Definition 5.7
• Theorem 5.10
• Definition A.1
• Theorem A.2