Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Benign Overfitting (ABO): Overparameterized RLS for Online Learning in Non-stationary Time-series

Luis Ontaneda Mijares, Nick Firoozye

TL;DR

Adaptive Benign Overfitting (ABO) extends online regression to the overparameterized, non-stationary regime by rephrasing Recursive Least Squares (RLS) with a numerically stable QR-based, orthogonal–triangular update. It couples Random Fourier Features with a sliding-window exponentially weighted scheme (QR-EWRLS), achieving $O(ND)$ per-update complexity and maintaining a minimum-norm solution in the $D\gg N$ regime, while tracking evolving data via a forgetting factor. The method reproduces benign overfitting phenomena, including double descent, and maintains bounded residuals and stable condition numbers in nonlinear time-series forecasting, with practical speedups of 20–40% on EUR/USD and electricity-load tasks compared to kernel baselines. By linking adaptive filtering, kernel approximation, and benign overfitting within a unified online framework, ABO enables scalable, stable high-dimensional online learning and suggests avenues for distributed ensemble approaches to further enhance scalability and robustness.

Abstract

Overparameterized models have recently challenged conventional learning theory by exhibiting improved generalization beyond the interpolation limit, a phenomenon known as benign overfitting. This work introduces Adaptive Benign Overfitting (ABO), extending the recursive least-squares (RLS) framework to this regime through a numerically stable formulation based on orthogonal-triangular updates. A QR-based exponentially weighted RLS (QR-EWRLS) algorithm is introduced, combining random Fourier feature mappings with forgetting-factor regularization to enable online adaptation under non-stationary conditions. The orthogonal decomposition prevents the numerical divergence associated with covariance-form RLS while retaining adaptability to evolving data distributions. Experiments on nonlinear synthetic time series confirm that the proposed approach maintains bounded residuals and stable condition numbers while reproducing the double-descent behavior characteristic of overparameterized models. Applications to forecasting foreign exchange and electricity demand show that ABO is highly accurate (comparable to baseline kernel methods) while achieving speed improvements of between 20 and 40 percent. The results provide a unified view linking adaptive filtering, kernel approximation, and benign overfitting within a stable online learning framework.

Adaptive Benign Overfitting (ABO): Overparameterized RLS for Online Learning in Non-stationary Time-series

TL;DR

Adaptive Benign Overfitting (ABO) extends online regression to the overparameterized, non-stationary regime by rephrasing Recursive Least Squares (RLS) with a numerically stable QR-based, orthogonal–triangular update. It couples Random Fourier Features with a sliding-window exponentially weighted scheme (QR-EWRLS), achieving per-update complexity and maintaining a minimum-norm solution in the regime, while tracking evolving data via a forgetting factor. The method reproduces benign overfitting phenomena, including double descent, and maintains bounded residuals and stable condition numbers in nonlinear time-series forecasting, with practical speedups of 20–40% on EUR/USD and electricity-load tasks compared to kernel baselines. By linking adaptive filtering, kernel approximation, and benign overfitting within a unified online framework, ABO enables scalable, stable high-dimensional online learning and suggests avenues for distributed ensemble approaches to further enhance scalability and robustness.

Abstract

Overparameterized models have recently challenged conventional learning theory by exhibiting improved generalization beyond the interpolation limit, a phenomenon known as benign overfitting. This work introduces Adaptive Benign Overfitting (ABO), extending the recursive least-squares (RLS) framework to this regime through a numerically stable formulation based on orthogonal-triangular updates. A QR-based exponentially weighted RLS (QR-EWRLS) algorithm is introduced, combining random Fourier feature mappings with forgetting-factor regularization to enable online adaptation under non-stationary conditions. The orthogonal decomposition prevents the numerical divergence associated with covariance-form RLS while retaining adaptability to evolving data distributions. Experiments on nonlinear synthetic time series confirm that the proposed approach maintains bounded residuals and stable condition numbers while reproducing the double-descent behavior characteristic of overparameterized models. Applications to forecasting foreign exchange and electricity demand show that ABO is highly accurate (comparable to baseline kernel methods) while achieving speed improvements of between 20 and 40 percent. The results provide a unified view linking adaptive filtering, kernel approximation, and benign overfitting within a stable online learning framework.
Paper Structure (56 sections, 2 theorems, 113 equations, 3 figures, 10 tables, 1 algorithm)

This paper contains 56 sections, 2 theorems, 113 equations, 3 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Benign Overfitting
  3. From Kernel Regression to Neural Tangent Kernels
  4. From Neural Tangent Kernels to Random Features
  5. Double Descent in RFFs
  6. Online Learning and Non-stationarity
  7. STFT and EWRLS with Random Fourier Features
  8. Computational Complexity and Spectral Deconvolution
  9. Extension Beyond the Time Domain
  10. Stability through QR Updates
  11. Contributions
  12. Related Work
  13. System Model
  14. Least squares via QR
  15. Random Fourier Features
  16. ...and 41 more sections

Key Result

Theorem 1

Let $\hat{\beta}_t$ denote the unique minimum-$\ell_2$-norm solution of the exponentially weighted sliding-window least-squares problem at time $t$, Let $\beta_t$ be the estimate produced by Algorithm 1. Assume exact arithmetic, admissible rank-preserving updates and downdates, and initialization $\beta_0 = \hat{\beta}_0$. Then, for all $t \ge 0$, the algorithmic iterate coincides with the minimu

Figures (3)

  • Figure 1: Illustration of the double-descent phenomenon in Random Fourier Features.
  • Figure 2: Double-descent behavior of test error
  • Figure 3: Walk-forward rolling validation with overlapping windows and strictly disjoint test windows, each preceded by batch initialization

Theorems & Definitions (4)

  • Theorem 1: Consistency with minimum-norm exponentially weighted LS
  • proof
  • Lemma 1: Weighted Downdate Consistency
  • proof