Generalization in Representation Models via Random Matrix Theory: Application to Recurrent Networks

TL;DR

The paper develops a unified random-matrix theory framework to analyze generalization for models that learn only a linear readout on top of fixed representations, including recurrent maps such as Echo State Networks. In the high-dimensional regime, it yields deterministic asymptotic expressions for the out-of-sample risk, decomposing it into bias and variance components, and provides explicit formulas in terms of fixed-point quantities and resolvents. Specializing to Linear ESNs, the authors show an equivalence to ridge regression with an exponentially memory-weighted input covariance, revealing a bias toward recent inputs and explaining the absence of double descent in this setting. Empirical results corroborate the theory, showing ESNs outperform ridge in low-data, short-memory scenarios and ridge regains superiority with more data or longer dependencies. Overall, the work offers a general theoretical framework for understanding generalization in overparameterized models with fixed representations and yields practical insights for designing recurrent representations.

Abstract

We first study the generalization error of models that use a fixed feature representation (frozen intermediate layers) followed by a trainable readout layer. This setting encompasses a range of architectures, from deep random-feature models to echo-state networks (ESNs) with recurrent dynamics. Working in the high-dimensional regime, we apply Random Matrix Theory to derive a closed-form expression for the asymptotic generalization error. We then apply this analysis to recurrent representations and obtain concise formula that characterize their performance. Surprisingly, we show that a linear ESN is equivalent to ridge regression with an exponentially time-weighted (''memory'') input covariance, revealing a clear inductive bias toward recent inputs. Experiments match predictions: ESNs win in low-sample, short-memory regimes, while ridge prevails with more data or long-range dependencies. Our methodology provides a general framework for analyzing overparameterized models and offers insights into the behavior of deep learning networks.

Figures (2)

• Figure 1: Comparison of test error curves for Linear ESNs and random projections as a function of $n/N$ . For random projections, the test error exhibits a clear double descent peak near the interpolation threshold ($n/N \approx 1$). In contrast, Linear ESNs show no such peak due to their effective low-rank feature covariance, as explained above.
• Figure 2: Comparison between ESNs and ridge regression. The test error is shown as a function of the sample size $N$ (y-axis) and the signal decay rate $\rho$ (x-axis), where $\boldsymbol{\theta}^\star = (\rho^t,\, t \in \{1,\dots,T\})$ for both models. Shaded regions indicate the standard deviation over multiple runs. The dashed line marks the separation threshold. ESNs exhibit superior performance in the limited-data, short-memory regime.

Theorems & Definitions (37)

• Definition 1: Noisy linear model
• Definition 2: Out-of-sample risk
• Definition 3: Concentrated random vector
• Theorem 1: Fixed Representation Generalization
• proof
• Remark 1: Special Cases
• Definition 4: Echo State Network
• Theorem 2: Linear ESN Generalization
• proof
• Remark 2: Difference with Ridge regression and Interpretation