State space models, emergence, and ergodicity: How many parameters are needed for stable predictions?

TL;DR

In the present work, it is shown that the problem of learning linear dynamical systems -- a simple instance of self-supervised learning -- exhibits a corresponding phase transition, and in this model, tasks exhibiting substantial long-range correlation require a certain critical number of parameters.

Abstract

How many parameters are required for a model to execute a given task? It has been argued that large language models, pre-trained via self-supervised learning, exhibit emergent capabilities such as multi-step reasoning as their number of parameters reach a critical scale. In the present work, we explore whether this phenomenon can analogously be replicated in a simple theoretical model. We show that the problem of learning linear dynamical systems -- a simple instance of self-supervised learning -- exhibits a corresponding phase transition. Namely, for every non-ergodic linear system there exists a critical threshold such that a learner using fewer parameters than said threshold cannot achieve bounded error for large sequence lengths. Put differently, in our model we find that tasks exhibiting substantial long-range correlation require a certain critical number of parameters -- a phenomenon akin to emergence. We also investigate the role of the learner's parametrization and consider a simple version of a linear dynamical system with hidden state -- an imperfectly observed random walk in $\mathbb{R}$. For this situation, we show that there exists no learner using a linear filter which can succesfully learn the random walk unless the filter length exceeds a certain threshold depending on the effective memory length and horizon of the problem.

Figures (1)

• Figure 1: We illustrate \ref{['thm:sfemergence']} by a simple numerical example of learning a $d_{\mathsf{X}}$-dimensional linear system $X_{t+1}=A_\star X_t+W_t$ with fewer than the required number of parameters and where $d_{\mathsf{X}}=7$. Namely, we only estimating the top-left $k\times k$ sub-matrix, $k\in [d_{\mathsf{X}}]$. We run the least squares estimator for samples drawn from $m \gg 2^{d_{\mathsf{X}}}$ many trajectories and vary the trajectory length, $T$. As predicted, as $T$ grows the risk diverges unless the parametrization is sufficiently high-dimensional, $k=4$, at which the point the risk drops to near zero and exhibits more stable behavior (note the logarithmic scale on the $y$-axis).

Theorems & Definitions (12)

• Theorem : Informal version of \ref{['thm:sfemergence']}
• Theorem : Informal version of \ref{['thm:stablepoldsthm']}
• Remark 2.1
• Theorem 3.1
• proof
• Proposition 4.1
• Theorem 4.1
• proof
• Lemma A.1
• proof