Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SPAT: Sensitivity-based Multihead-attention Pruning on Time Series Forecasting Models

Suhan Guo, Jiahong Deng, Mengjun Yi, Furao Shen, Jian Zhao

TL;DR

SPAT tackles the computational burden of attention in multivariate time series forecasting by pruning entire MHA modules via a dispersion-based sensitivity metric, SEND, computed during pretraining. The method yields substantial efficiency gains (FLOPs and parameter reductions) while preserving or improving forecasting accuracy and zero-shot transfer capabilities. Across eight real-world datasets, SPAT-pruned models outperform many lightweight, Mamba-based, and LLM-based SOTA methods in most settings, underscoring the value of retaining only the most impactful attention mechanisms. This work demonstrates that targeted, structure-preserving pruning can deliver practical, hardware-friendly improvements for time-series forecasting with Transformer architectures.

Abstract

Attention-based architectures have achieved superior performance in multivariate time series forecasting but are computationally expensive. Techniques such as patching and adaptive masking have been developed to reduce their sizes and latencies. In this work, we propose a structured pruning method, SPAT ($\textbf{S}$ensitivity $\textbf{P}$runer for $\textbf{At}$tention), which selectively removes redundant attention mechanisms and yields highly effective models. Different from previous approaches, SPAT aims to remove the entire attention module, which reduces the risk of overfitting and enables speed-up without demanding specialized hardware. We propose a dynamic sensitivity metric, $\textbf{S}$ensitivity $\textbf{E}$nhanced $\textbf{N}$ormalized $\textbf{D}$ispersion (SEND) that measures the importance of each attention module during the pre-training phase. Experiments on multivariate datasets demonstrate that SPAT-pruned models achieve reductions of 2.842% in MSE, 1.996% in MAE, and 35.274% in FLOPs. Furthermore, SPAT-pruned models outperform existing lightweight, Mamba-based and LLM-based SOTA methods in both standard and zero-shot inference, highlighting the importance of retaining only the most effective attention mechanisms. We have made our code publicly available https://anonymous.4open.science/r/SPAT-6042.

SPAT: Sensitivity-based Multihead-attention Pruning on Time Series Forecasting Models

TL;DR

SPAT tackles the computational burden of attention in multivariate time series forecasting by pruning entire MHA modules via a dispersion-based sensitivity metric, SEND, computed during pretraining. The method yields substantial efficiency gains (FLOPs and parameter reductions) while preserving or improving forecasting accuracy and zero-shot transfer capabilities. Across eight real-world datasets, SPAT-pruned models outperform many lightweight, Mamba-based, and LLM-based SOTA methods in most settings, underscoring the value of retaining only the most impactful attention mechanisms. This work demonstrates that targeted, structure-preserving pruning can deliver practical, hardware-friendly improvements for time-series forecasting with Transformer architectures.

Abstract

Attention-based architectures have achieved superior performance in multivariate time series forecasting but are computationally expensive. Techniques such as patching and adaptive masking have been developed to reduce their sizes and latencies. In this work, we propose a structured pruning method, SPAT (ensitivity runer for tention), which selectively removes redundant attention mechanisms and yields highly effective models. Different from previous approaches, SPAT aims to remove the entire attention module, which reduces the risk of overfitting and enables speed-up without demanding specialized hardware. We propose a dynamic sensitivity metric, ensitivity nhanced ormalized ispersion (SEND) that measures the importance of each attention module during the pre-training phase. Experiments on multivariate datasets demonstrate that SPAT-pruned models achieve reductions of 2.842% in MSE, 1.996% in MAE, and 35.274% in FLOPs. Furthermore, SPAT-pruned models outperform existing lightweight, Mamba-based and LLM-based SOTA methods in both standard and zero-shot inference, highlighting the importance of retaining only the most effective attention mechanisms. We have made our code publicly available https://anonymous.4open.science/r/SPAT-6042.
Paper Structure (30 sections, 18 equations, 5 figures, 8 tables, 1 algorithm)

This paper contains 30 sections, 18 equations, 5 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Transformer-based forecasters
  4. Light-weight forecasters
  5. Attention pruning
  6. Methods
  7. Preliminary
  8. SEND
  9. Ranking and Pruning
  10. Experiments
  11. Experimental set-up
  12. Results
  13. Original vs. Pruned Models
  14. Pruned vs. Benchmark Models
  15. Zero-shot Inference
  16. ...and 15 more sections

Figures (5)

  • Figure 1: Attention score visualization: (a) Degenerate vs. (b) Effective. Degenerate attention is equivalent to scaled identity mapping.
  • Figure 2: Illustration of attention-based models (a) and pruning algorithm workflow (b). Solid arrows indicate feedforward; dashed arrows indicate backpropagation. MultiHead is defined in Eq.\ref{['eq:multihead']} and detailed in Appendix Figure\ref{['fig:attention']}.
  • Figure 3: Model efficiency comparison with average results over 4 forecasting horizons on Traffic. The batch size is set to 1. Figure \ref{['fig:lookback_gflops']} shows that PatchTST FLOPs vary with lookback length (bottom x-axis); iTransformer FLOPs vary with channel count (top x-axis).
  • Figure 4: Influence of lookback window lengths $L$
  • Figure 5: The MultiHead attention mechanism structure. $T(\cdot)$ and $\pi(\cdot)$ denotes the transpose and softmax operations. The upper part describes the detailed operations in each attention head, while the lower part only marks the input and output of the current head.