Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Asymptotic Theory of $K$-fold Cross-validation in Lasso and the validity of Bootstrap

Mayukh Choudhury, Debraj Das

TL;DR

The paper analyzes the Lasso estimator with a penalty chosen by $K$-fold cross-validation in a heteroscedastic linear model, establishing $n^{1/2}$-consistency but failure of variable selection consistency for the CV-based penalty. It derives the limiting behavior of the cross-validated penalty $oldsymbol{ exthat{ extlambda}}_{n,K}$, showing $n^{-1/2}oldsymbol{ exthat{ extlambda}}_{n,K} o_d oldsymbol{ extLambda}_{ ty,K}$, and proves that its asymptotic distribution is complex so that standard Gaussian inference is inappropriate. To enable valid inference, the authors introduce a Perturbation Bootstrap approach that recalculates the CV penalty on bootstrap samples and prove its conditional distribution converges to the same limit as the original estimator, establishing bootstrap validity. They validate the theory with simulations and a real-data example (prostate cancer dataset), demonstrating accurate finite-sample coverage and practical usefulness of bootstrap-based inference for CV-selected Lasso parameters.

Abstract

Least absolute shrinkage and selection operator or Lasso is one of the widely used regularization methods in regression. Statisticians usually implement Lasso in practice by choosing the penalty parameter in a data-dependent way, the most popular being the $K-$fold cross-validation (or $K-$fold CV). However, inferential properties, such as the variable selection consistency and $n^{1/2}-$consistency, of the $K-$fold CV based Lasso estimator and validity of the Bootstrap approximation are still unknown. In this paper, we consider the heteroscedastic linear regression model and show only under some moment type conditions that the Lasso estimator with $K$-fold CV based penalty is $n^{1/2}-$consistent, but not variable selection consistent. Additionally, we establish the validity of Bootstrap in approximating the distribution of the $K-$fold CV based Lasso estimator. Therefore, our results theoretically justify the use of $K-$fold CV based Lasso estimator to perform statistical inference in linear regression. We validate our Bootstrap method for the $K-$fold CV based Lasso estimator in finite samples based on simulations. We also implement our Bootstrap based inference on a real data set.

Asymptotic Theory of $K$-fold Cross-validation in Lasso and the validity of Bootstrap

TL;DR

The paper analyzes the Lasso estimator with a penalty chosen by -fold cross-validation in a heteroscedastic linear model, establishing -consistency but failure of variable selection consistency for the CV-based penalty. It derives the limiting behavior of the cross-validated penalty , showing , and proves that its asymptotic distribution is complex so that standard Gaussian inference is inappropriate. To enable valid inference, the authors introduce a Perturbation Bootstrap approach that recalculates the CV penalty on bootstrap samples and prove its conditional distribution converges to the same limit as the original estimator, establishing bootstrap validity. They validate the theory with simulations and a real-data example (prostate cancer dataset), demonstrating accurate finite-sample coverage and practical usefulness of bootstrap-based inference for CV-selected Lasso parameters.

Abstract

Least absolute shrinkage and selection operator or Lasso is one of the widely used regularization methods in regression. Statisticians usually implement Lasso in practice by choosing the penalty parameter in a data-dependent way, the most popular being the fold cross-validation (or fold CV). However, inferential properties, such as the variable selection consistency and consistency, of the fold CV based Lasso estimator and validity of the Bootstrap approximation are still unknown. In this paper, we consider the heteroscedastic linear regression model and show only under some moment type conditions that the Lasso estimator with -fold CV based penalty is consistent, but not variable selection consistent. Additionally, we establish the validity of Bootstrap in approximating the distribution of the fold CV based Lasso estimator. Therefore, our results theoretically justify the use of fold CV based Lasso estimator to perform statistical inference in linear regression. We validate our Bootstrap method for the fold CV based Lasso estimator in finite samples based on simulations. We also implement our Bootstrap based inference on a real data set.

Paper Structure

This paper contains 24 sections, 23 theorems, 120 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. The setup
  3. Our Contributions
  4. Our techniques
  5. Related literature
  6. Organization of the paper
  7. Preliminaries and regularity conditions
  8. Failure of VSC of $\hat{\bm{\beta}}_n(\hat{\lambda}_{n,K}$)
  9. $n^{1/2}$-Consistency of $\hat{\bm{\beta}}_n(\hat{\lambda}_{n,K})$
  10. Validity of Bootstrap approximation
  11. Simulation Study
  12. Real Data Analysis
  13. Proofs
  14. Measurability of Argmin Maps
  15. Requisite Lemmas related to Section \ref{['sec:crossv']}
  16. ...and 9 more sections

Key Result

Proposition 3.1

Suppose that the assumptions (C.1),(C.2) and (C.3) are true. Then we have

Figures (2)

  • Figure 1: Schematic representation of our findings. ($\hat{\bm{\beta}}_n^*(\hat{\lambda}_{n,K}^*)$ and $\tilde{\bm{\beta}}_n(\hat{\lambda}_{n,K})$ are defined in section \ref{['sec:bootstrap']} and other notations are defined in Table \ref{['tab:unified']})
  • Figure 2: Uniqueness of $\hat{\Lambda}_{\infty,K}$ over different covariance structures $\bm{S}$.

Theorems & Definitions (36)

  • Proposition 3.1
  • Theorem 3.1
  • Proposition 4.1
  • Theorem 4.1
  • Theorem 4.2
  • Remark 4.1
  • Remark 4.2
  • Remark 4.3
  • Theorem 5.1
  • Definition 8.1
  • ...and 26 more