Why Do Transformers Fail to Forecast Time Series In-Context?

TL;DR

This work analyzes why Transformers struggle on time-series forecasting by studying Linear Self-Attention (LSA) within an In-Context Learning framework for Auto-Regressive (AR) processes. By Hankelizing the input, the authors show one-layer LSA lives in a restricted cubic feature space and, even with infinite context, asymptotically collapses to the last $p$ lags, making it no better than linear regression (LR) for AR($p$) forecasting. They prove a strict finite-sample gap between LSA and LR, quantified via a positive semidefinite Schur-complement $\Delta_n$, with $\Delta_n = \tfrac{1}{n} B_p + o(1/n)$ under Gaussianity, and extend the result to general linear stationary processes through cumulant-based expansions. Chain-of-Thought rollout further shows exponential error growth and collapse to the mean, making CoT inferior to Bayes/LR at every horizon. Experiments with synthetic AR data validate the theory, showing LSA tracks AR($p$) but does not exceed the linear baseline, while CoT suffers from rapid error accumulation, supporting a call for architectures beyond naive attention for TSF.

Abstract

Time series forecasting (TSF) remains a challenging and largely unsolved problem in machine learning, despite significant recent efforts leveraging Large Language Models (LLMs), which predominantly rely on Transformer architectures. Empirical evidence consistently shows that even powerful Transformers often fail to outperform much simpler models, e.g., linear models, on TSF tasks; however, a rigorous theoretical understanding of this phenomenon remains limited. In this paper, we provide a theoretical analysis of Transformers' limitations for TSF through the lens of In-Context Learning (ICL) theory. Specifically, under AR($p$) data, we establish that: (1) Linear Self-Attention (LSA) models $\textit{cannot}$ achieve lower expected MSE than classical linear models for in-context forecasting; (2) as the context length approaches to infinity, LSA asymptotically recovers the optimal linear predictor; and (3) under Chain-of-Thought (CoT) style inference, predictions collapse to the mean exponentially. We empirically validate these findings through carefully designed experiments. Our theory not only sheds light on several previously underexplored phenomena but also offers practical insights for designing more effective forecasting architectures. We hope our work encourages the broader research community to revisit the fundamental theoretical limitations of TSF and to critically evaluate the direct application of increasingly sophisticated architectures without deeper scrutiny.

Figures (2)

• Figure 1: Experimental results. (a--b) Predictions under Teacher-Forcing (TF) and Chain-of-Thought (CoT). (c--d) Cumulative MSE for TF and CoT rollouts. (e--f) Scaling experiments varying the history length and the number of LSA layers. Overall, LSA tracks AR($p$) but never surpasses the OLS baseline, confirming its representational limits.
• Figure 2: Experimental results on comparison of LSA and Softmax Attention. (a--b) Predictions under Teacher-Forcing (TF) and Chain-of-Thought (CoT). (c--d) Cumulative MSE for TF and CoT rollouts. Overall, both LSA and Softmax Attention tracks AR($p$) but never surpass the OLS baseline. Moreover, Softmax Attention is slightly better than LSA.

Theorems & Definitions (95)

• Definition 2.1: Time Series
• Definition 2.2: AR($p$) Process hamilton2020time
• Definition 2.3: Linear Self-Attention (LSA)
• Definition 2.4: $L$-Layer LSA-Only Transformer
• Definition 2.5: Hankel Matrix
• Lemma 3.1: Cubic lifting for one-layer LSA
• Proposition 3.2: Information monotonicity
• Proposition 3.3: Asymptotic collapse of LSA features
• Theorem 3.4: Strict finite-sample gap: AR($p$)
• proof : Proof Sketch