A Shrinkage Likelihood Ratio Test for High-Dimensional Subgroup Analysis with a Logistic-Normal Mixture Model

TL;DR

The study tackles the challenge of testing for a subgroup with a differential treatment effect when subgroup classification parameters are not identified under the null, which yields irregular likelihood-based tests. It introduces a Shrinkage Likelihood Ratio Test (SLRT) based on a penalized likelihood with an $L_1$ penalty on the classification parameters within a logistic-normal mixture model, and leverages a reparameterization to achieve a simple half-chi-square null distribution even in high-dimensional settings. Theoretical results show consistency of the penalized estimators and that SLRT converges to $\chi^2_1/2 + \chi^2_0/2$ under $H_0$, with the penalty driving the high-dimensional nuisance parameters toward zero under the null. Monte Carlo simulations and real data analysis (ACTG175) demonstrate good finite-sample size control, improved power in high-dimensional settings, and practical applicability, offering a scalable tool for robust subgroup discovery in precision medicine. The approach provides tractable critical values and favorable computation, enabling reliable subgroup inference with many covariates and potential extensions to more complex data regimes.

Abstract

In subgroup analysis, testing the existence of a subgroup with a differential treatment effect serves as protection against spurious subgroup discovery. Despite its importance, this hypothesis testing possesses a complicated nature: parameter characterizing subgroup classification is not identified under the null hypothesis of no subgroup. Due to this irregularity, the existing methods have the following two limitations. First, the asymptotic null distribution of test statistics often takes an intractable form, which necessitates computationally demanding resampling methods to calculate the critical value. Second, the dimension of personal attributes characterizing subgroup membership is not allowed to be of high dimension. To solve these two problems simultaneously, this study develops a shrinkage likelihood ratio test for the existence of a subgroup using a logistic-normal mixture model. The proposed test statistics are built on a modified likelihood function that shrinks possibly high-dimensional unidentified parameters toward zero under the null hypothesis while retaining power under the alternative. This shrinkage helps handle the irregularity and restore the simple chi-square-type asymptotics even under the high-dimensional regime.

Theorems & Definitions (18)

• Proposition 1
• Proposition 2
• Proposition 3
• proof : Proof of Proposition \ref{['consistency']}
• proof : Proof of Proposition \ref{['quad approx']}
• proof : Proof of Proposition \ref{['asy null dist']}
• Lemma 1
• proof : Proof of Lemma \ref{['compact']}
• Lemma 2
• proof : Proof of Lemma \ref{['consistency lemma']}