Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OLinear: A Linear Model for Time Series Forecasting in Orthogonally Transformed Domain

Wenzhen Yue, Yong Liu, Haoxuan Li, Hao Wang, Xianghua Ying, Ruohao Guo, Bowei Xing, Ji Shi

TL;DR

OLinear introduces a two-part framework for time-series forecasting in an orthogonally transformed domain. OrthoTrans creates a dataset-adaptive orthogonal basis from the temporal Pearson correlation matrix to decorrelate signals, while NormLin provides a normalized linear layer that captures multivariate dependencies with high rank and efficiency. Together, these components enable a simple, linear-based forecaster that achieves state-of-the-art accuracy across 24 benchmarks and 140 forecasting tasks, and can plug into Transformer-based forecasters to boost performance and efficiency. The work demonstrates strong generalization, scalability to large models, and robustness to training data size, suggesting practical impact for diverse real-world forecasting needs. The approach also offers insights into representation learning under decorrelated domains and may inform future token-dependency learning beyond self-attention.

Abstract

This paper presents $\mathbf{OLinear}$, a $\mathbf{linear}$-based multivariate time series forecasting model that operates in an $\mathbf{o}$rthogonally transformed domain. Recent forecasting models typically adopt the temporal forecast (TF) paradigm, which directly encode and decode time series in the time domain. However, the entangled step-wise dependencies in series data can hinder the performance of TF. To address this, some forecasters conduct encoding and decoding in the transformed domain using fixed, dataset-independent bases (e.g., sine and cosine signals in the Fourier transform). In contrast, we utilize $\mathbf{OrthoTrans}$, a data-adaptive transformation based on an orthogonal matrix that diagonalizes the series' temporal Pearson correlation matrix. This approach enables more effective encoding and decoding in the decorrelated feature domain and can serve as a plug-in module to enhance existing forecasters. To enhance the representation learning for multivariate time series, we introduce a customized linear layer, $\mathbf{NormLin}$, which employs a normalized weight matrix to capture multivariate dependencies. Empirically, the NormLin module shows a surprising performance advantage over multi-head self-attention, while requiring nearly half the FLOPs. Extensive experiments on 24 benchmarks and 140 forecasting tasks demonstrate that OLinear consistently achieves state-of-the-art performance with high efficiency. Notably, as a plug-in replacement for self-attention, the NormLin module consistently enhances Transformer-based forecasters. The code and datasets are available at https://anonymous.4open.science/r/OLinear

OLinear: A Linear Model for Time Series Forecasting in Orthogonally Transformed Domain

TL;DR

OLinear introduces a two-part framework for time-series forecasting in an orthogonally transformed domain. OrthoTrans creates a dataset-adaptive orthogonal basis from the temporal Pearson correlation matrix to decorrelate signals, while NormLin provides a normalized linear layer that captures multivariate dependencies with high rank and efficiency. Together, these components enable a simple, linear-based forecaster that achieves state-of-the-art accuracy across 24 benchmarks and 140 forecasting tasks, and can plug into Transformer-based forecasters to boost performance and efficiency. The work demonstrates strong generalization, scalability to large models, and robustness to training data size, suggesting practical impact for diverse real-world forecasting needs. The approach also offers insights into representation learning under decorrelated domains and may inform future token-dependency learning beyond self-attention.

Abstract

This paper presents , a -based multivariate time series forecasting model that operates in an rthogonally transformed domain. Recent forecasting models typically adopt the temporal forecast (TF) paradigm, which directly encode and decode time series in the time domain. However, the entangled step-wise dependencies in series data can hinder the performance of TF. To address this, some forecasters conduct encoding and decoding in the transformed domain using fixed, dataset-independent bases (e.g., sine and cosine signals in the Fourier transform). In contrast, we utilize , a data-adaptive transformation based on an orthogonal matrix that diagonalizes the series' temporal Pearson correlation matrix. This approach enables more effective encoding and decoding in the decorrelated feature domain and can serve as a plug-in module to enhance existing forecasters. To enhance the representation learning for multivariate time series, we introduce a customized linear layer, , which employs a normalized weight matrix to capture multivariate dependencies. Empirically, the NormLin module shows a surprising performance advantage over multi-head self-attention, while requiring nearly half the FLOPs. Extensive experiments on 24 benchmarks and 140 forecasting tasks demonstrate that OLinear consistently achieves state-of-the-art performance with high efficiency. Notably, as a plug-in replacement for self-attention, the NormLin module consistently enhances Transformer-based forecasters. The code and datasets are available at https://anonymous.4open.science/r/OLinear
Paper Structure (62 sections, 1 theorem, 22 equations, 17 figures, 38 tables)

This paper contains 62 sections, 1 theorem, 22 equations, 17 figures, 38 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Transformed domain in time series forecasting
  4. Time series forecasters
  5. Preliminaries
  6. Method
  7. Problem formulation
  8. Overall architecture
  9. Orthogonal transformation (OrthoTrans)
  10. Discussion
  11. Representation learning
  12. CSL and NormLin
  13. ISL
  14. Discussion
  15. Experiments
  16. ...and 47 more sections

Key Result

Theorem 1

We assume that $\mathbf{x} \sim \mathcal{N} \left ( \mu _ {\mathbf{x}}, \Sigma _{\mathbf{x}} \right )$, $y \sim \mathcal{N} \left ( \mu _y, \sigma _y^2 \right )$, and the joint distribution $[\mathbf{x},y] \sim \mathcal{N} \left ( \left [ \mu _{\mathbf{x}}, \mu _y \right ], \right )$. Here, $\

Figures (17)

  • Figure 1: Overall structure of OLinear. The orthogonal matrix derived from the series' temporal Pearson correlation matrix transforms the series into the new feature domain with removed sequential correlations. The cross-series learners and intra-series learners are then employed for robust representation learning. Specifically, we introduce the NormLin layer, whose weight matrix is processed by the Softplus function and row-wise L1 normalization, to capture multivariate correlations.
  • Figure 2: Comparison between the time and transformed domains. (1) The series exhibits strong consistency in the transformed domain, which is desirable for forecasting. (2) Sequence correlations are effectively reduced in the transformed domain. (3) Periodicity can be clearly reflected in the correlation matrix. (4) The new domain typically exhibits energy compaction, with only a few dominant dimensions. More cases are shown in Figure \ref{['fig:time_trans_corr_Q']}.
  • Figure 3: Model efficiency comparison. Bubble areas represent GPU memory usage during training, scaled independently in the two subfigures. Resource footprint data are from Table \ref{['tab:GPU']}.
  • Figure 4: Visualization of the forecasting results of OLinear, demonstrating great accuracy.
  • Figure 5: The learned NormLin weights exhibit similarity to the multivariate correlation matrix (after Softmax).
  • ...and 12 more figures

Theorems & Definitions (2)

  • Theorem 1: Expected value of $y$
  • proof