Universal Redundancies in Time Series Foundation Models

TL;DR

This work analyzes transformer-based Time Series Foundation Models (TSFMs) to forecast unseen series in a zero-shot setting. It reveals universal redundancies in intermediate layers, especially the middle layers, and introduces a mechanistic interpretability toolbox including residual-stream ablations and direct logit attribution. A kernel-regression perspective of attention shows cross-attention behaves like a Nadaraya-Watson estimator, with sharp heads driving context parroting and seasonality bias; the authors propose an intrinsic head-pruning strategy based on stable rank $\text{sr}(A) = \|A\|_F^2/\|A\|_2^2$, preserving forecast structure even after ablating up to about $28\%$ of heads across models on GIFT-Eval and related benchmarks. These findings highlight universal redundancies in TSFMs and offer practical compression strategies for zero-shot forecasting, backed by an open-source toolbox for mechanistic interpretability.

Abstract

Time Series Foundation Models (TSFMs) leverage extensive pretraining to accurately predict unseen time series during inference, without the need for task-specific fine-tuning. Through large-scale evaluations on standard benchmarks, we find that leading transformer-based TSFMs exhibit redundant components in their intermediate layers. We introduce a set of tools for mechanistic interpretability of TSFMs, including ablations of specific components and direct logit attribution on the residual stream. Our findings are consistent across several leading TSFMs with diverse architectures, and across a diverse set of real-world and synthetic time-series datasets. We discover that all models in our study are robust to ablations of entire layers. Furthermore, we develop a theoretical framework framing transformers as kernel regressors, motivating a purely intrinsic strategy for ablating heads based on the stable rank of the per-head projection matrices. Using this approach, we uncover the specific heads responsible for degenerate phenomena widely observed in TSFMs, such as parroting of motifs from the context and seasonality bias. Our study sheds light on the universal properties of this emerging class of architectures for continuous-time sequence modeling.

Figures (23)

• Figure 1: Middle layers are highly redundant in their contribution to the residual stream: For an example forecasting task (A), we investigate the logit maps produced by direct logit attribution (DLA) on the residual stream (B) after each layer in the Chronos decoder. As seen in (C), the middle layers qualitatively perform very similar updates. We quantify this observation (D) by showing that these middle layers introduce uncertainty, measured as an increase in the entropy of the resulting distribution over tokens. Specifically, we compute $\frac{1}{T}\sum_{t=0}^{T} \sigma(H^{(\ell)}W_{\text{out}})$ for $401$ distinct forecast tasks; we highlight the median for each layer. We also measure the similarity among head outputs for the heads in each layer, and we observe a higher average entropic rank (Appendix \ref{['section:entropic_rank_discussion']}) in the middle layers (E), further suggesting redundancy in the middle layers. Appendix \ref{['section:more_residual_stream_measurements']} presents more examples of measurements on the residual stream.
• Figure 2: Middle Layers are more ablateable than early and late layers: Spearman distance ($1 - \rho$) between the original model predictions and the predictions with ablations of entire layers (i.e. All Heads and the MLP) for groups of layers. Across all the TSFMs we evaluate, the middle layers show greater ablatability, suggesting redundant components, whereas the first and last layers are the most important to preserving the model's performance. Appendix \ref{['section:more_ablation_results']} presents more measurements of the depthwise importance of components.
• Figure 3: TSFMs have redundant components: Across all the leading models in our study, we identify layers with redundant components. We report the MASE geometric mean of the models with ablations against the original model on GIFT-Eval.
• Figure 4: TSFMs exhibit significant variation in sensitivity to component ablations: When ablating components in contiguous layers, (A) Chronos Bolt shows extreme resilience to the MLP ablations, maintaining its performance even after half of them are removed. (B) Toto shows much greater sensitivity.
• Figure 5: Sharp heads as a mechanism for context parroting: Attention scores between prediction and context timesteps for Chronos show clear delineation of low entropy"sharp" (A) and high entropy "diffuse" heads (B). Sharp heads correspond to kernel regression with a small bandwidth, whereas diffuse heads are highly redundant and can ablated without affecting the context parroting (C) and (D). Moreover, removing a single sharp head can break the parroting (Fig. \ref{['fig:chronos_failure_ablate_sharp_heads']}). Appendix \ref{['section:extended_discussion_attention_kernel_regression']} presents further discussion.

Theorems & Definitions (5)

• Proposition 4.1: Informal
• Definition 5.1: Stable Rank
• Definition 5.2: heads@1pp
• Proposition 3.1
• proof