New explanations and inference for least angle regression

TL;DR

This paper reframes least angle regression (LAR) as an inferential procedure on a population path Lar( X, μ ), introducing the population quantities $C_k$ and equi-angular vectors $\mathbf{a}_k$ to interpret how predictors enter the model. It develops a rigorous asymptotic theory for the LAR sample path, proving consistency to the population path and deriving limiting distributions for the step correlations, with a formal bootstrap method to quantify uncertainty and determine a termination point $\bar{m}$. The work provides practical tools, including bootstrap-based confidence intervals for both step correlations and step coefficients and visualizations of the inferred LAR path, validated through simulations and real-data examples (face temperature and diabetes datasets). Overall, the framework yields a principled stopping rule and reliable uncertainty quantification for LAR, with potential extensions to high-dimensional regimes and connections to Lasso-like methods. The results deepen understanding of LAR mechanics, enabling more robust interpretation and application in regression settings where variable selection and inference are intertwined.

Abstract

Efron et al. (2004) introduced least angle regression (LAR) as an algorithm for linear predictions, intended as an alternative to forward selection with connections to penalized regression. However, LAR has remained somewhat of a "black box," where some basic behavioral properties of LAR output are not well understood, including an appropriate termination point for the algorithm. We provide a novel framework for inference with LAR, which also allows LAR to be understood from new perspectives with several newly developed mathematical properties. The LAR algorithm at a data level can viewed as estimating a population counterpart "path" that organizes a response mean along regressor variables which are ordered according to a decreasing series of population "correlation" parameters; such parameters are shown to have meaningful interpretations for explaining variable contributions whereby zero correlations denote unimportant variables. In the output of LAR, estimates of all non-zero population correlations turn out to have independent normal distributions for use in inference, while estimates of zero-valued population correlations have a certain non-normal joint distribution. These properties help to provide a formal rule for stopping the LAR algorithm. While the standard bootstrap for regression can fail for LAR, a modified bootstrap provides a practical and formally justified tool for interpreting the entrance of variables and quantifying uncertainty in estimation. The LAR inference method is studied through simulation and illustrated with data examples.

Figures (9)

• Figure 1: Column $\mathbf{x}_{j_{k+1}}$ enters on step $k+1$, while $\mathbf{x}_j$ does not enter. The dashed lines denote either $(\mathbf{c}_k)_{j_{k+1}} - \gamma (\mathbf{w}_k)_{j_{k+1}}$ or $(\mathbf{c}_k)_j - \gamma (\mathbf{w}_k)_j$, while the solid lines denote $C_k - \gamma A_k$ and $-C_k + \gamma A_k$. The horizontal and vertical bands illustrate condition (M2) of Theorem \ref{['thm:larconsistency']}.
• Figure 2: Sample path $\operatorname{Lar}(\mathbf{X},\mathbf{y})$ on face temperature data described in Section \ref{['sec:facetemp']}.
• Figure 3: Inferred LAR path for the face temperature data based on $\bar{m} = 6$.
• Figure 4: Left: Estimation of $m$. Right: Bootstrap probability of active set membership for each variable index.
• Figure 5: Inferred LAR path from the diabetes data based on $\bar{m} = 5$.

Theorems & Definitions (20)

• Lemma 2
• Definition 4
• Proposition 5
• Lemma 6
• Lemma 7
• Lemma 8
• Lemma 9
• Remark 10
• Remark 11
• Remark 12