Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SCFormer: Structured Channel-wise Transformer with Cumulative Historical State for Multivariate Time Series Forecasting

Shiwei Guo, Ziang Chen, Yupeng Ma, Yunfei Han, Yi Wang

TL;DR

SCFormer addresses limitations of channel-wise Transformers in multivariate time series forecasting by enforcing temporal consistency on all linear transformations and by incorporating long-range historical information through HiPPO-based cumulative historical state. It unifies the look-back window with a cumulative history and uses structured linear transforms (triangular matrices or 1D convolutions) to preserve temporal causality across Q, K, V and feed-forward layers. The approach yields significant accuracy gains across real-world datasets, with notable parameter efficiency, and ablations confirm the critical roles of temporal constraints, HiPPO memory, and look-back integration. The findings highlight the practical value of memory-augmented, temporally constrained transformers for robust, scalable forecasting in multivariate time series tasks.

Abstract

The Transformer model has shown strong performance in multivariate time series forecasting by leveraging channel-wise self-attention. However, this approach lacks temporal constraints when computing temporal features and does not utilize cumulative historical series effectively.To address these limitations, we propose the Structured Channel-wise Transformer with Cumulative Historical state (SCFormer). SCFormer introduces temporal constraints to all linear transformations, including the query, key, and value matrices, as well as the fully connected layers within the Transformer. Additionally, SCFormer employs High-order Polynomial Projection Operators (HiPPO) to deal with cumulative historical time series, allowing the model to incorporate information beyond the look-back window during prediction. Extensive experiments on multiple real-world datasets demonstrate that SCFormer significantly outperforms mainstream baselines, highlighting its effectiveness in enhancing time series forecasting. The code is publicly available at https://github.com/ShiweiGuo1995/SCFormer

SCFormer: Structured Channel-wise Transformer with Cumulative Historical State for Multivariate Time Series Forecasting

TL;DR

SCFormer addresses limitations of channel-wise Transformers in multivariate time series forecasting by enforcing temporal consistency on all linear transformations and by incorporating long-range historical information through HiPPO-based cumulative historical state. It unifies the look-back window with a cumulative history and uses structured linear transforms (triangular matrices or 1D convolutions) to preserve temporal causality across Q, K, V and feed-forward layers. The approach yields significant accuracy gains across real-world datasets, with notable parameter efficiency, and ablations confirm the critical roles of temporal constraints, HiPPO memory, and look-back integration. The findings highlight the practical value of memory-augmented, temporally constrained transformers for robust, scalable forecasting in multivariate time series tasks.

Abstract

The Transformer model has shown strong performance in multivariate time series forecasting by leveraging channel-wise self-attention. However, this approach lacks temporal constraints when computing temporal features and does not utilize cumulative historical series effectively.To address these limitations, we propose the Structured Channel-wise Transformer with Cumulative Historical state (SCFormer). SCFormer introduces temporal constraints to all linear transformations, including the query, key, and value matrices, as well as the fully connected layers within the Transformer. Additionally, SCFormer employs High-order Polynomial Projection Operators (HiPPO) to deal with cumulative historical time series, allowing the model to incorporate information beyond the look-back window during prediction. Extensive experiments on multiple real-world datasets demonstrate that SCFormer significantly outperforms mainstream baselines, highlighting its effectiveness in enhancing time series forecasting. The code is publicly available at https://github.com/ShiweiGuo1995/SCFormer
Paper Structure (26 sections, 16 equations, 5 figures, 4 tables)

This paper contains 26 sections, 16 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Improving Quadratic Computation Cost for Transformer
  4. Transformer with Patching
  5. Channel-wise Transformer
  6. Cumulative Historical Utilization
  7. METHOD
  8. Cumulative Historical State
  9. Triangular Matrix and Temporal Constraint
  10. Convolutional Self-attention
  11. Instance Normalized and Loss Function
  12. Experiments
  13. Datasets and implementation
  14. Datasets
  15. Implementation
  16. ...and 11 more sections

Figures (5)

  • Figure 1: (a) Structured linear transformation (Right) vs. Linear transformation (Left). The temporal constraint of the series is preserved by setting the weights of successor elements to 0, ensuring that these elements do not influence the current element. (b) Markov forecasting process (Bottom) vs. Forecasting process with cumulative historical state (Top). A model using only the look-back as input essentially operates as a Markov process, where forecasting is the modeling of the transition matrix. In contrast, our model leverages the cumulative historical state to retain the state of a more complete historical series, with forecasting corresponding to the modeling of the emission matrix.
  • Figure 2: Overall structure of SCFormer. For forecasting at a given moment, the model first computes the cumulative historical state via HiPPO and combines it with the look-back as the final input. Then, temporal constraints are applied to the feature computation through multiple structured linear transformations in the channel-wise Transformer. (A) Cumulative historical state via HiPPO; (B) Embedding; (C) Structured channel-wise self-attention; (D) Structured feed-forward layer.
  • Figure 3: The computation process of HiPPO: The coefficients $c$ are obtained by projecting the sequence $f(t)$ onto an orthogonal polynomial basis under the metric $u$. These coefficients represent the optimal parameters when approximating the sequence $f(t)$ using the orthogonal polynomial basis. HiPPO enables efficient recursive computation through state-space equations.
  • Figure 4: The effect of look-back length: The 720 window size look-back (green) significantly reduces the prediction error compared to the 96 window size (blue). (a) The MSE of SCFormer-conv on Solar-Energy. (b) The MSE of SCFormer-triangular on Solar-Energy. (c) The MAE of SCFormer-conv on Traffic. (d) The MAE of SCFormer-triangular on Traffic. (e) The MSE of SCFormer-conv on ETTm1. (f) The MSE of SCFormer-triangular on ETTm1.
  • Figure 5: (a) Channels(multivariate) correlations: Left-Top: the future correlations of Traffic; Left-Bottom: the attention scores of Traffic; Right-Top: the future correlations of Solar-Energy; Right-Bottom: the attention scores of Solar-Energy. (b) Example visualization of iTransformer on ECL. (c) Example visualization of SCFormer-triangular on ECL.