Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Universality in block dependent linear models with applications to nonparametric regression

Samriddha Lahiry, Pragya Sur

TL;DR

This work extends high-dimensional universality results to block-dependent covariate designs in the proportional regime by showing that the optimal empirical risk and the estimation risk for convex penalties (Lasso and ridge) converge to the corresponding Gaussian-design limits, even when covariates exhibit a block structure. The authors formalize a block-dependent sub-Gaussian design class and derive fixed-point characterizations that govern the exact asymptotic risks in nonparametric regression settings obtained via basis expansions (e.g., penalized additive models and scalar-on-function regression). A novel leave-d-out technique combined with Stein/CGMT-inspired arguments establishes compactness and universality of both the optimum and the optimizer under these dependencies. The results enable precise risk predictions for high-dimensional nonparametric regressions in applications like GWAS and functional data analysis, with experiments showing universality emerges at moderate sample sizes.

Abstract

Over the past decade, characterizing the exact asymptotic risk of regularized estimators in high-dimensional regression has emerged as a popular line of work. This literature considers the proportional asymptotics framework, where the number of features and samples both diverge, at a rate proportional to each other. Substantial work in this area relies on Gaussianity assumptions on the observed covariates. Further, these studies often assume the design entries to be independent and identically distributed. Parallel research investigates the universality of these findings, revealing that results based on the i.i.d.~Gaussian assumption extend to a broad class of designs, such as i.i.d.~sub-Gaussians. However, universality results examining dependent covariates so far focused on correlation-based dependence or a highly structured form of dependence, as permitted by right rotationally invariant designs. In this paper, we break this barrier and study a dependence structure that in general falls outside the purview of these established classes. We seek to pin down the extent to which results based on i.i.d.~Gaussian assumptions persist. We identify a class of designs characterized by a block dependence structure that ensures the universality of i.i.d.~Gaussian-based results. We establish that the optimal values of the regularized empirical risk and the risk associated with convex regularized estimators, such as the Lasso and ridge, converge to the same limit under block dependent designs as they do for i.i.d.~Gaussian entry designs. Our dependence structure differs significantly from correlation-based dependence, and enables, for the first time, asymptotically exact risk characterization in prevalent nonparametric regression problems in high dimensions. Finally, we illustrate through experiments that this universality becomes evident quite early, even for relatively moderate sample sizes.

Universality in block dependent linear models with applications to nonparametric regression

TL;DR

This work extends high-dimensional universality results to block-dependent covariate designs in the proportional regime by showing that the optimal empirical risk and the estimation risk for convex penalties (Lasso and ridge) converge to the corresponding Gaussian-design limits, even when covariates exhibit a block structure. The authors formalize a block-dependent sub-Gaussian design class and derive fixed-point characterizations that govern the exact asymptotic risks in nonparametric regression settings obtained via basis expansions (e.g., penalized additive models and scalar-on-function regression). A novel leave-d-out technique combined with Stein/CGMT-inspired arguments establishes compactness and universality of both the optimum and the optimizer under these dependencies. The results enable precise risk predictions for high-dimensional nonparametric regressions in applications like GWAS and functional data analysis, with experiments showing universality emerges at moderate sample sizes.

Abstract

Over the past decade, characterizing the exact asymptotic risk of regularized estimators in high-dimensional regression has emerged as a popular line of work. This literature considers the proportional asymptotics framework, where the number of features and samples both diverge, at a rate proportional to each other. Substantial work in this area relies on Gaussianity assumptions on the observed covariates. Further, these studies often assume the design entries to be independent and identically distributed. Parallel research investigates the universality of these findings, revealing that results based on the i.i.d.~Gaussian assumption extend to a broad class of designs, such as i.i.d.~sub-Gaussians. However, universality results examining dependent covariates so far focused on correlation-based dependence or a highly structured form of dependence, as permitted by right rotationally invariant designs. In this paper, we break this barrier and study a dependence structure that in general falls outside the purview of these established classes. We seek to pin down the extent to which results based on i.i.d.~Gaussian assumptions persist. We identify a class of designs characterized by a block dependence structure that ensures the universality of i.i.d.~Gaussian-based results. We establish that the optimal values of the regularized empirical risk and the risk associated with convex regularized estimators, such as the Lasso and ridge, converge to the same limit under block dependent designs as they do for i.i.d.~Gaussian entry designs. Our dependence structure differs significantly from correlation-based dependence, and enables, for the first time, asymptotically exact risk characterization in prevalent nonparametric regression problems in high dimensions. Finally, we illustrate through experiments that this universality becomes evident quite early, even for relatively moderate sample sizes.
Paper Structure (26 sections, 13 theorems, 177 equations, 2 figures)

This paper contains 26 sections, 13 theorems, 177 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Outline of the results
  3. Notations
  4. Examples of block dependent designs
  5. Penalized additive regression model
  6. Penalized functional regression
  7. Block dependent covariates in GWAS
  8. Preliminaries
  9. Assumptions
  10. Choice of penalty function
  11. Fixed point equations for the estimation risk
  12. Main result
  13. Illustration
  14. Penalized additive regression model
  15. Penalized functional regression
  16. ...and 11 more sections

Key Result

Theorem 4.1

Suppose the design matrix $\mathcal{X}=X\Lambda^{1/2}$ satisfies Assumption ass_1 and $\beta_0$ satisfies Assumption ass_2. Further $\mathcal{G}=G\Lambda^{1/2}$, where $G \in \mathbb{R}^{n \times p}$ contains i.i.d. Gaussian entries, and that $\psi$ is a bounded differentiable function with bounded Further if we assume Assumption ass_3 in addition to Assumptions ass_1-ass_2, then we have

Figures (2)

  • Figure 1: Estimation risk in the penalized additive model (see Section \ref{['add_ex']}) averaged over 50 iterations with $n=200,p_0=30$, and $d=10$. The red curves correspond to the risk $\|\beta^L_{\mathcal{X}}-\beta_0\|^2$ and $\|\beta^R_{\mathcal{X}}-\beta_0\|^2$ for the Lasso and the ridge respectively, while the corresponding risks for the Gaussian problem $\|\beta^L_{\mathcal{G}}-\beta_0\|^2$ and $\|\beta^R_{\mathcal{G}}-\beta_0\|^2$ are overlaid as black dots.
  • Figure 2: Estimation risk in the penalized functional regression model (see Section \ref{['fr_ex']}) averaged over 50 iterations with $n=500,p_0=30$, and $d=10$. The red curves corresponds to the risk $\|\beta^L_{\mathcal{X}}-\beta_0\|^2$ and $\|\beta^R_{\mathcal{X}}-\beta_0\|^2$ for Lasso and risk respectively, while the corresponding risks for the Gaussian $\|\beta^L_{\mathcal{G}}-\beta_0\|^2$ and $\|\beta^R_{\mathcal{G}}-\beta_0\|^2$ are overlaid as black dots.

Theorems & Definitions (27)

  • Definition 2.1
  • Remark 2.1
  • Remark 3.1
  • Theorem 4.1
  • Corollary 4.1
  • Theorem 4.2
  • Corollary 4.2
  • Corollary 4.3
  • Corollary 4.4
  • Remark 4.1
  • ...and 17 more