Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lasso and Partially-Rotated Designs

Rares-Darius Buhai

TL;DR

This work investigates sparse linear regression with semirandom designs, focusing on Lasso performance when a subset of design columns may be arbitrarily correlated. It introduces partially-rotated designs and the deterministic restricted normalized orthogonality (RNO) property, proving that RNO implies RE constants for all subsets of the rotated part, thereby making the Lasso fast-rate bound effectively independent of the arbitrary columns. The main result shows that for X partially rotated with respect to S and n large enough, the Lasso prediction error scales as O(σ^2 k log d / (λ_min n)) where λ_min is the smallest eigenvalue of X_{supp(β)}^T X_{supp(β)} / n, and the bound depends only on the well-conditioned part X_S. The analysis leverages sparsification via approximate Carathéodory, a link between RNO and RE, and a probabilistic argument that partially-rotated matrices satisfy RNO with high probability. Collectively, the results broaden conditions under which Lasso attains fast rates in settings with correlated, semirandom design columns, highlighting robustness to outlier-like columns outside the secret support.

Abstract

We consider the sparse linear regression model $\mathbf{y} = X β+\mathbf{w}$, where $X \in \mathbb{R}^{n \times d}$ is the design, $β\in \mathbb{R}^{d}$ is a $k$-sparse secret, and $\mathbf{w} \sim N(0, I_n)$ is the noise. Given input $X$ and $\mathbf{y}$, the goal is to estimate $β$. In this setting, the Lasso estimate achieves prediction error $O(k \log d / γn)$, where $γ$ is the restricted eigenvalue (RE) constant of $X$ with respect to $\mathrm{support}(β)$. In this paper, we introduce a new $\textit{semirandom}$ family of designs -- which we call $\textit{partially-rotated}$ designs -- for which the RE constant with respect to the secret is bounded away from zero even when a subset of the design columns are arbitrarily correlated among themselves. As an example of such a design, suppose we start with some arbitrary $X$, and then apply a random rotation to the columns of $X$ indexed by $\mathrm{support}(β)$. Let $λ_{\min}$ be the smallest eigenvalue of $\frac{1}{n} X_{\mathrm{support}(β)}^\top X_{\mathrm{support}(β)}$, where $X_{\mathrm{support}(β)}$ is the restriction of $X$ to the columns indexed by $\mathrm{support}(β)$. In this setting, our results imply that Lasso achieves prediction error $O(k \log d / λ_{\min} n)$ with high probability. This prediction error bound is independent of the arbitrary columns of $X$ not indexed by $\mathrm{support}(β)$, and is as good as if all of these columns were perfectly well-conditioned. Technically, our proof reduces to showing that matrices with a certain deterministic property -- which we call $\textit{restricted normalized orthogonality}$ (RNO) -- lead to RE constants that are independent of a subset of the matrix columns. This property is similar but incomparable with the restricted orthogonality condition of [CT05].

Lasso and Partially-Rotated Designs

TL;DR

This work investigates sparse linear regression with semirandom designs, focusing on Lasso performance when a subset of design columns may be arbitrarily correlated. It introduces partially-rotated designs and the deterministic restricted normalized orthogonality (RNO) property, proving that RNO implies RE constants for all subsets of the rotated part, thereby making the Lasso fast-rate bound effectively independent of the arbitrary columns. The main result shows that for X partially rotated with respect to S and n large enough, the Lasso prediction error scales as O(σ^2 k log d / (λ_min n)) where λ_min is the smallest eigenvalue of X_{supp(β)}^T X_{supp(β)} / n, and the bound depends only on the well-conditioned part X_S. The analysis leverages sparsification via approximate Carathéodory, a link between RNO and RE, and a probabilistic argument that partially-rotated matrices satisfy RNO with high probability. Collectively, the results broaden conditions under which Lasso attains fast rates in settings with correlated, semirandom design columns, highlighting robustness to outlier-like columns outside the secret support.

Abstract

We consider the sparse linear regression model , where is the design, is a -sparse secret, and is the noise. Given input and , the goal is to estimate . In this setting, the Lasso estimate achieves prediction error , where is the restricted eigenvalue (RE) constant of with respect to . In this paper, we introduce a new family of designs -- which we call designs -- for which the RE constant with respect to the secret is bounded away from zero even when a subset of the design columns are arbitrarily correlated among themselves. As an example of such a design, suppose we start with some arbitrary , and then apply a random rotation to the columns of indexed by . Let be the smallest eigenvalue of , where is the restriction of to the columns indexed by . In this setting, our results imply that Lasso achieves prediction error with high probability. This prediction error bound is independent of the arbitrary columns of not indexed by , and is as good as if all of these columns were perfectly well-conditioned. Technically, our proof reduces to showing that matrices with a certain deterministic property -- which we call (RNO) -- lead to RE constants that are independent of a subset of the matrix columns. This property is similar but incomparable with the restricted orthogonality condition of [CT05].
Paper Structure (21 sections, 16 theorems, 48 equations)

This paper contains 21 sections, 16 theorems, 48 equations.

Table of Contents

  1. Introduction
  2. Lasso
  3. Notation
  4. Main result: partially-rotated designs
  5. Our framework: restricted normalized orthogonality
  6. RNO and other conditions
  7. Discussion: restricted eigenvalue constant
  8. Related work
  9. Organization
  10. Techniques
  11. Attempt via Hölder's inequality
  12. Result for sparse linear combinations
  13. Sparsification
  14. Sparsification with $\ell_1$-norm error
  15. Proof that RNO implies RE
  16. ...and 6 more sections

Key Result

Lemma 1.3

Let $\bm{R} \in \varmathbb{R}^{n \times n}$ be a random matrix such that, for a fixed unit vector $v \in {\varmathbb R}^d$, $\bm{R} v$ is distributed uniformly over the unit sphere. (E.g., take $\bm{R}$ to be a random orthogonal matrix.) Then, for any matrix $X \in {\varmathbb R}^{n \times d}$ and a

Theorems & Definitions (31)

  • Definition 1.1: Restricted eigenvalue constant
  • Definition 1.2: Partially-rotated matrix
  • Lemma 1.3: Random rotation implies partial rotation
  • Lemma 1.4: Sub-Gaussianity implies partial rotation
  • Theorem 1.5: Main result
  • Corollary 1.6: Main result for Lasso
  • proof
  • Definition 1.9: Restricted normalized orthogonality
  • Proposition 1.10: RNO implies RE
  • Lemma 1.11: Partially-rotated matrices satisfy RNO
  • ...and 21 more