Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benign Overfitting in Time Series Linear Models with Over-Parameterization

Shogo Nakakita, Masaaki Imaizumi

TL;DR

This work analyzes benign overfitting in an over-parameterized time-series linear regression setting using a minimum-norm interpolation estimator. It derives non-asymptotic risk bounds under a coherence condition on temporal covariances, introducing effective ranks and showing the risk depends on the product of temporal covariance matrices; importantly, interpolation induces implicit temporal decorrelation that aligns the dependent-case risk with the i.i.d. benchmark in favorable regimes. The authors define benign covariance and prove a convergence rate $R(\widehat{\beta}) = O_P(\tau_n + \nu_n \eta_n)$, where $\nu_n = \|\Xi_{0,n}^{-1}\Upsilon_n\|$, and illustrate the results with Gaussian examples including separable ARMA, non-separable ARMA, and time-varying regression models. This advances benign overfitting theory to dependent, non-sparse time-series, with implications for high-dimensional, non-sparse modeling of complex temporal data.

Abstract

The success of large-scale models in recent years has increased the importance of statistical models with numerous parameters. Several studies have analyzed over-parameterized linear models with high-dimensional data, which may not be sparse; however, existing results rely on the assumption of sample independence. In this study, we analyze a linear regression model with dependent time-series data in an over-parameterized setting. We consider an estimator using interpolation and develop a theory for the excess risk of the estimator. Then, we derive non-asymptotic risk bounds for the estimator for cases with dependent data. This analysis reveals that the coherence of the temporal covariance plays a key role; the risk bound is influenced by the product of temporal covariance matrices at different time steps. Moreover, we show the convergence rate of the risk bound and demonstrate that it is also influenced by the coherence of the temporal covariance. Finally, we provide several examples of specific dependent processes applicable to our setting.

Benign Overfitting in Time Series Linear Models with Over-Parameterization

TL;DR

This work analyzes benign overfitting in an over-parameterized time-series linear regression setting using a minimum-norm interpolation estimator. It derives non-asymptotic risk bounds under a coherence condition on temporal covariances, introducing effective ranks and showing the risk depends on the product of temporal covariance matrices; importantly, interpolation induces implicit temporal decorrelation that aligns the dependent-case risk with the i.i.d. benchmark in favorable regimes. The authors define benign covariance and prove a convergence rate , where , and illustrate the results with Gaussian examples including separable ARMA, non-separable ARMA, and time-varying regression models. This advances benign overfitting theory to dependent, non-sparse time-series, with implications for high-dimensional, non-sparse modeling of complex temporal data.

Abstract

The success of large-scale models in recent years has increased the importance of statistical models with numerous parameters. Several studies have analyzed over-parameterized linear models with high-dimensional data, which may not be sparse; however, existing results rely on the assumption of sample independence. In this study, we analyze a linear regression model with dependent time-series data in an over-parameterized setting. We consider an estimator using interpolation and develop a theory for the excess risk of the estimator. Then, we derive non-asymptotic risk bounds for the estimator for cases with dependent data. This analysis reveals that the coherence of the temporal covariance plays a key role; the risk bound is influenced by the product of temporal covariance matrices at different time steps. Moreover, we show the convergence rate of the risk bound and demonstrate that it is also influenced by the coherence of the temporal covariance. Finally, we provide several examples of specific dependent processes applicable to our setting.
Paper Structure (33 sections, 24 theorems, 143 equations)

This paper contains 33 sections, 24 theorems, 143 equations.

Table of Contents

  1. Introduction
  2. Related studies
  3. Organization
  4. Notation
  5. Setting and assumption
  6. Gaussian process with spatio-temporal covariance
  7. Stochastic regression problem and interpolation estimator
  8. Effective ranks and numbers
  9. Non-asymptotic risk analysis
  10. Assumption: coherent temporal covariance
  11. Upper bound on the excess risk
  12. Lower bound on the excess risk
  13. Benign covariance: convergence rate analysis
  14. Preparation
  15. Convergence rate
  16. ...and 18 more sections

Key Result

Theorem 1

Consider the interpolation estimator $\widehat{\beta}$ in def:interpolator for the stochastic regression problem def:model. Suppose that Assumption asmp:coherence and $\lambda_{n+1}>0$ holds. Then, there exist constants $b=b(\epsilon)\ge 1$ and $c=c(\epsilon)\ge 1$ dependent only on $\epsilon$ such with probability at least $1-\delta-c\exp(-n/c)$.

Theorems & Definitions (47)

  • Example 1: vector ARMA process
  • Remark 1
  • Definition 1: effective rank
  • Theorem 1
  • Remark 2: implicit temporal decorrelation
  • Theorem 2
  • Definition 2: benign covariance
  • Theorem 3
  • Proposition 4: convergence rate
  • Lemma 5
  • ...and 37 more