Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Universality of estimators for high-dimensional linear models with block dependency

Toshiki Tsuda, Masaaki Imaizumi

TL;DR

This paper tackles the challenge of universality for estimators in high-dimensional linear models when covariates exhibit block dependence rather than Gaussianity. It introduces a generalized Lindeberg principle to handle block structure and develops an explicit error bound that links the universality gap to the block size $d$ and smoothness properties of the loss and regularizer. The main results show that, under suitable growth conditions and moment assumptions, the distribution of estimators under block-dependent covariates matches that under Gaussian covariates with the same moments, including for robust estimators like absolute loss and Huber loss as well as standard regularized methods such as Lasso and Ridge. The work also provides precise application results and numerical experiments demonstrating universality even when block sizes are as large as the parameter dimension $p$, highlighting the practical impact for high-dimensional inference under realistic dependency structures.

Abstract

We study the universality property of estimators for high-dimensional linear models, which implies that the distribution of estimators is independent of whether the covariates follow a Gaussian distribution. Recent developments in high-dimensional statistics typically require covariates to strictly follow a Gaussian distribution to precisely characterize the properties of estimators. To relax this Gaussianity requirement, the existing literature has examined conditions under which estimators achieve universality. In particular, independence among the elements of the high-dimensional covariates has played a critical role. In this study, we focus on high-dimensional linear models with covariates exhibiting block dependence, where covariate elements can only be dependent within each block, and show that estimators for such models retain universality. Specifically, we prove that the distribution of estimators with Gaussian covariates can be approximated by the distribution of estimators with non-Gaussian covariates having the same moments under block dependence. To establish this result, we develop a generalized Lindeberg principle suitable for handling block dependencies and derive new error bounds for correlated covariate elements. We further demonstrate the universality result across several different estimators.

Universality of estimators for high-dimensional linear models with block dependency

TL;DR

This paper tackles the challenge of universality for estimators in high-dimensional linear models when covariates exhibit block dependence rather than Gaussianity. It introduces a generalized Lindeberg principle to handle block structure and develops an explicit error bound that links the universality gap to the block size and smoothness properties of the loss and regularizer. The main results show that, under suitable growth conditions and moment assumptions, the distribution of estimators under block-dependent covariates matches that under Gaussian covariates with the same moments, including for robust estimators like absolute loss and Huber loss as well as standard regularized methods such as Lasso and Ridge. The work also provides precise application results and numerical experiments demonstrating universality even when block sizes are as large as the parameter dimension , highlighting the practical impact for high-dimensional inference under realistic dependency structures.

Abstract

We study the universality property of estimators for high-dimensional linear models, which implies that the distribution of estimators is independent of whether the covariates follow a Gaussian distribution. Recent developments in high-dimensional statistics typically require covariates to strictly follow a Gaussian distribution to precisely characterize the properties of estimators. To relax this Gaussianity requirement, the existing literature has examined conditions under which estimators achieve universality. In particular, independence among the elements of the high-dimensional covariates has played a critical role. In this study, we focus on high-dimensional linear models with covariates exhibiting block dependence, where covariate elements can only be dependent within each block, and show that estimators for such models retain universality. Specifically, we prove that the distribution of estimators with Gaussian covariates can be approximated by the distribution of estimators with non-Gaussian covariates having the same moments under block dependence. To establish this result, we develop a generalized Lindeberg principle suitable for handling block dependencies and derive new error bounds for correlated covariate elements. We further demonstrate the universality result across several different estimators.

Paper Structure

This paper contains 55 sections, 21 theorems, 128 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Notation
  3. Setup with block sparsity
  4. Setup: estimator with convex risk minimization
  5. Basic assumptions
  6. Dependence structure
  7. Main result of universality
  8. Gaussian analogy
  9. Universality of empirical risk
  10. Universality of estimator
  11. Outline of proof with Lindeberg principle
  12. Overview
  13. Bound for $IV$: Lindeberg principle with block dependence
  14. Generalized Lindeberg principle
  15. Bounding sum of block size
  16. ...and 40 more sections

Key Result

Theorem 1

Suppose Assumptions ass::regime, ass::loss and ass::moment hold. Also, suppose that the distribution of $X_1,...,X_n$ is $d$-block dependent. Then, for sufficiently large $n$, there exists some $C_0 =C_0(\tau,q,M)>0$ such that the following hold: for any $\mathcal{S}_{n}\subset[-L_{n}, L_{n}]^{p}$ w Here $K_{g}:=1 +\max_{\ell\in[0:3]}\|g^{(\ell)}\|_{\infty}$ and $r_R(L_n)$ is defined by where we

Figures (2)

  • Figure 1: Q-Q plots of the estimation errors. The horizontal axis is for the Gaussian design, and the vertical axis is for the $t$‑distributed design. We set $n=100,p=50$.
  • Figure 2: Q-Q plots of the estimation errors. The horizontal axis is for the Gaussian design, and the vertical axis is for the $t$‑distributed design. We set $n=200,p=100$.

Theorems & Definitions (45)

  • Example 1: Linear regression with convex losses
  • Definition 1: Block dependence
  • Remark 1: In related studies
  • Theorem 1
  • Corollary 2
  • Theorem 3
  • Corollary 4
  • Lemma 5
  • Lemma 6
  • proof
  • ...and 35 more