Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Asymptotics of Linear Regression with Linearly Dependent Data

Behrad Moniri, Hamed Hassani

TL;DR

This work addresses the high-dimensional ridge regression problem with linearly dependent covariates exhibiting spatio-temporal covariance. It establishes a Gaussian universality principle, showing that the asymptotic estimation error is invariant to non-Gaussian perturbations in the covariates when mean and covariance are preserved, and derives a precise fixed-point characterization of the error via spectral data of the covariate transforms. The authors show that the limiting error is governed by a scalar function $m(\lambda;\gamma)$ and its fixed-point solution $\kappa$, enabling efficient computation of bias, variance, and total error, and they identify the optimal ridge parameter $\lambda_\star=\sigma_\varepsilon^2\gamma/\alpha^2$ independent of dependence structure. They also analyze how dependence affects overparameterization and the double descent phenomenon, supported by simulations that align with theoretical predictions. Overall, the results provide actionable insights for high-dimensional regression under dependence, including precise error characterizations and guidance on regularization in structured covariate settings.

Abstract

In this paper we study the asymptotics of linear regression in settings with non-Gaussian covariates where the covariates exhibit a linear dependency structure, departing from the standard assumption of independence. We model the covariates using stochastic processes with spatio-temporal covariance and analyze the performance of ridge regression in the high-dimensional proportional regime, where the number of samples and feature dimensions grow proportionally. A Gaussian universality theorem is proven, demonstrating that the asymptotics are invariant under replacing the non-Gaussian covariates with Gaussian vectors preserving mean and covariance, for which tools from random matrix theory can be used to derive precise characterizations of the estimation error. The estimation error is characterized by a fixed-point equation involving the spectral properties of the spatio-temporal covariance matrices, enabling efficient computation. We then study optimal regularization, overparameterization, and the double descent phenomenon in the context of dependent data. Simulations validate our theoretical predictions, shedding light on how dependencies influence estimation error and the choice of regularization parameters.

Asymptotics of Linear Regression with Linearly Dependent Data

TL;DR

This work addresses the high-dimensional ridge regression problem with linearly dependent covariates exhibiting spatio-temporal covariance. It establishes a Gaussian universality principle, showing that the asymptotic estimation error is invariant to non-Gaussian perturbations in the covariates when mean and covariance are preserved, and derives a precise fixed-point characterization of the error via spectral data of the covariate transforms. The authors show that the limiting error is governed by a scalar function and its fixed-point solution , enabling efficient computation of bias, variance, and total error, and they identify the optimal ridge parameter independent of dependence structure. They also analyze how dependence affects overparameterization and the double descent phenomenon, supported by simulations that align with theoretical predictions. Overall, the results provide actionable insights for high-dimensional regression under dependence, including precise error characterizations and guidance on regularization in structured covariate settings.

Abstract

In this paper we study the asymptotics of linear regression in settings with non-Gaussian covariates where the covariates exhibit a linear dependency structure, departing from the standard assumption of independence. We model the covariates using stochastic processes with spatio-temporal covariance and analyze the performance of ridge regression in the high-dimensional proportional regime, where the number of samples and feature dimensions grow proportionally. A Gaussian universality theorem is proven, demonstrating that the asymptotics are invariant under replacing the non-Gaussian covariates with Gaussian vectors preserving mean and covariance, for which tools from random matrix theory can be used to derive precise characterizations of the estimation error. The estimation error is characterized by a fixed-point equation involving the spectral properties of the spatio-temporal covariance matrices, enabling efficient computation. We then study optimal regularization, overparameterization, and the double descent phenomenon in the context of dependent data. Simulations validate our theoretical predictions, shedding light on how dependencies influence estimation error and the choice of regularization parameters.

Paper Structure

This paper contains 30 sections, 4 theorems, 74 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Related Work
  4. Main Results
  5. Universality
  6. Proof Sketch.
  7. Precise Asymptotics
  8. Proof Sketch.
  9. Optimal Regularization
  10. Proof.
  11. Special Cases
  12. Case 1: $B^\top B = I_d$ and $A^\top A = I_n$
  13. Case 2: $A^\top A = I_n$
  14. Case 3: $B^\top B = I_d$
  15. Examples
  16. ...and 15 more sections

Key Result

theorem 4

Let $G= [g_1, \dots, g_n]^\top \in \mathbb{R}^{n \times d}$ have entries drawn i.i.d. from Gaussian distribution with the same mean and variance as $\mathcal{D}_z$, and let $\tilde{X} = A G B$ and $\tilde{y} = \tilde{X} \beta_\star + \varepsilon$. Under Conditions def:covariates-cond:random_effect, with probability $1 - o(1)$, under the condition that $\mathcal{D}_z$ is sub-Gaussian.

Figures (4)

  • Figure 1: The estimation error $R(\hat{\beta})$ as a function of $\gamma$ with $\mu_A = 1/3\, \delta_{1} + 1/3\, \delta_{2} + 1/3\, \delta_{3}$, $\mu_B = 1/2\, \delta_{1} + 1/2\, \delta_{2}$, $\sigma_\varepsilon = 0.2$, and $\alpha = 0.7$. We set $\lambda = 0.03$ (Left) and $\lambda = \gamma$ (Right). In these plots, the solid lines are the theoretical predictions using Corollary \ref{['cor:final']} and the dots are numerical predictions, averaged over 50 trials.
  • Figure 2: The estimation error as a function of $\omega$ for Example 2 in Section \ref{['sec:examples']} as a function of $\omega$ with $\sigma_\varepsilon =1, \alpha = 1$, $\gamma = 2$ and $\lambda = \lambda_\star$.
  • Figure 3: The estimation error as a function of $\lambda$ in the same setting of Figure \ref{['fig:as_dep']} with $\sigma_\varepsilon =1, \alpha = 1$, and $\gamma = 0.2$.
  • Figure 4: The estimation error, bias, and variance of Example 2 in Section \ref{['sec:examples']} with $\sigma_\varepsilon = 1, \alpha = 1,$ and $\gamma = 0.2$ as a function of $\gamma$ for $\omega = 0.8$ (Left) and $\omega = 0.2$ (Right).

Theorems & Definitions (4)

  • theorem 4: Gaussian Universality
  • theorem 6
  • corollary 1
  • theorem 7