Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Post-selection inference for penalized M-estimators via score thinning

Ronan Perry, Snigdha Panigrahi, Daniela Witten

TL;DR

This work addresses invalid inference after selecting a model with sparsity-inducing M-estimators by introducing score thinning, which decomposes the score into two approximately normal, independent components that separately govern selection and inference. By either adding noise to the penalty or to the data (noising the samples), the authors show that model selection and standard downstream inference can be made asymptotically valid without bespoke selective procedures, under milder distributional assumptions and an independence premise. The framework extends to generalized linear models with $L_1$ penalties and is demonstrated through simulations and a network-based study, illustrating nominal coverage and practical interval widths. Overall, the method enables practitioners to use standard inference pipelines after data-driven model selection, offering a simple, scalable alternative to existing bespoke selective-inference techniques with broad applicability.

Abstract

We consider inference for M-estimators after model selection using a sparsity-inducing penalty. While existing methods for this task require bespoke inference procedures, we propose a simpler approach, which relies on two insights: (i) adding and subtracting carefully-constructed noise to a Gaussian random variable with unknown mean and known variance leads to two \emph{independent} Gaussian random variables; and (ii) both the selection event resulting from penalized M-estimation, and the event that a standard (non-selective) confidence interval for an M-estimator covers its target, can be characterized in terms of an approximately normal ``score variable". We combine these insights to show that -- when the noise is chosen carefully -- there is asymptotic independence between the model selected using a noisy penalized M-estimator, and the event that a standard (non-selective) confidence interval on noisy data covers the selected parameter. Therefore, selecting a model via penalized M-estimation (e.g. \verb=glmnet= in \verb=R=) on noisy data, and then conducting \emph{standard} inference on the selected model (e.g. \verb=glm= in \verb=R=) using noisy data, yields valid inference: \emph{no bespoke methods are required}. Our results require independence of the observations, but only weak distributional requirements. We apply the proposed approach to conduct inference on the association between sex and smoking in a social network.

Post-selection inference for penalized M-estimators via score thinning

TL;DR

This work addresses invalid inference after selecting a model with sparsity-inducing M-estimators by introducing score thinning, which decomposes the score into two approximately normal, independent components that separately govern selection and inference. By either adding noise to the penalty or to the data (noising the samples), the authors show that model selection and standard downstream inference can be made asymptotically valid without bespoke selective procedures, under milder distributional assumptions and an independence premise. The framework extends to generalized linear models with penalties and is demonstrated through simulations and a network-based study, illustrating nominal coverage and practical interval widths. Overall, the method enables practitioners to use standard inference pipelines after data-driven model selection, offering a simple, scalable alternative to existing bespoke selective-inference techniques with broad applicability.

Abstract

We consider inference for M-estimators after model selection using a sparsity-inducing penalty. While existing methods for this task require bespoke inference procedures, we propose a simpler approach, which relies on two insights: (i) adding and subtracting carefully-constructed noise to a Gaussian random variable with unknown mean and known variance leads to two \emph{independent} Gaussian random variables; and (ii) both the selection event resulting from penalized M-estimation, and the event that a standard (non-selective) confidence interval for an M-estimator covers its target, can be characterized in terms of an approximately normal ``score variable". We combine these insights to show that -- when the noise is chosen carefully -- there is asymptotic independence between the model selected using a noisy penalized M-estimator, and the event that a standard (non-selective) confidence interval on noisy data covers the selected parameter. Therefore, selecting a model via penalized M-estimation (e.g. \verb=glmnet= in \verb=R=) on noisy data, and then conducting \emph{standard} inference on the selected model (e.g. \verb=glm= in \verb=R=) using noisy data, yields valid inference: \emph{no bespoke methods are required}. Our results require independence of the observations, but only weak distributional requirements. We apply the proposed approach to conduct inference on the association between sex and smoking in a social network.
Paper Structure (39 sections, 12 theorems, 139 equations, 4 figures)

This paper contains 39 sections, 12 theorems, 139 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Existing approaches for inference after model selection
  3. Contributions of this paper
  4. Valid post-selection inference via a score variable
  5. M-estimation
  6. The score variable
  7. Approximate normality of the thinned score variable
  8. Asymptotically valid post-selection inference
  9. Post-selection inference by adding a noisy penalty
  10. Post-selection inference by noising the samples
  11. Application to generalized linear models
  12. Experimental results
  13. Simulation studies
  14. Application to principal components network regression
  15. Discussion
  16. ...and 24 more sections

Key Result

Lemma 1

Suppose that $C \subseteq \mathbb{R}^{2p}$ is the union of $K$ measurable and almost surely disjoint convex sets. Under Condition cond:berry_esseen, for any $\Delta \geq 0$,

Figures (4)

  • Figure 1: Outcomes are generated from a linear model with i.i.d. Gaussian noise whose variance is unknown. Inference is conducted on the linear model selected by $L_1$-penalized ordinary least squares. Results are aggregated across $1000$ repetitions. Since the classical approach fails to attain valid coverage, we omit it from the center and right-hand panels.
  • Figure 2: We simulate outcomes from a logistic regression model. Inference is conducted on the model selected using $L_1$-penalized logistic regression. Confidence intervals are computed for all coefficients in the selected model. Results are aggregated across $1000$ repetitions. Since the classical approach fails to attain valid coverage, we omit it from the center and right-hand panels.
  • Figure 3: Outcomes are generated from a linear model with equicorrelated non-Gaussian noise. Inference is conducted in the linear model selected by $L_1$-penalized linear regression. Results are aggregated across $5000$ repetitions. Since the classical approach fails to attain valid coverage, we omit it from the center and right-hand panels.
  • Figure 4: (Left): the friendship network of students, colored by recorded sex. (Middle): the friendship network of students, colored by their reported tobacco usage $Y$. (Right): the noisy outcomes constructed by adding and subtracting Gaussian noise to the outcomes $Y$, i.e., $Y_i + \bar{W}_{n, i}$ and $Y_i - \bar{W}_{n, i}$. These sets are not independent, but selection and inference using them are asymptotically independent.

Theorems & Definitions (21)

  • Lemma 1
  • Lemma 2
  • Proposition 3
  • Lemma 4
  • Lemma 5
  • Theorem 6: Post-selection inference via a noisy penalty
  • Corollary 7: Valid post-selection inference via a noisy response
  • Proposition 8
  • Theorem A.1: Berry-Esseen raic_multivariate_2019
  • Lemma A.2
  • ...and 11 more