A Lightweight Sparse Interaction Network for Time Series Forecasting

TL;DR

This paper tackles long-horizon time series forecasting by replacing costly self-attention with explicit, sparse temporal interactions implemented in a lightweight linear model called LSINet. The core ideas are Multihead Sparse Interaction Mechanism (MSIM), which learns important time-step connections via a sparsity-induced Bernoulli process, and Shared Interactions Learning (SIL), which captures repetitive interaction patterns across samples and variables. An auxiliary self-adaptive sparse regularization loss (ASRL) and a time-invariant mixing component enable robust, efficient learning, while Patch Encoding converts history into patches for structured processing. Empirically, LSINet delivers higher forecasting accuracy and lower compute/memory overhead than both transformer-based and prior linear TSF models across multiple datasets, validating the practical value of explicit sparse temporal interactions in a scalable, linear framework.

Abstract

Recent work shows that linear models can outperform several transformer models in long-term time-series forecasting (TSF). However, instead of explicitly performing temporal interaction through self-attention, linear models implicitly perform it based on stacked MLP structures, which may be insufficient in capturing the complex temporal dependencies and their performance still has potential for improvement. To this end, we propose a Lightweight Sparse Interaction Network (LSINet) for TSF task. Inspired by the sparsity of self-attention, we propose a Multihead Sparse Interaction Mechanism (MSIM). Different from self-attention, MSIM learns the important connections between time steps through sparsity-induced Bernoulli distribution to capture temporal dependencies for TSF. The sparsity is ensured by the proposed self-adaptive regularization loss. Moreover, we observe the shareability of temporal interactions and propose to perform Shared Interaction Learning (SIL) for MSIM to further enhance efficiency and improve convergence. LSINet is a linear model comprising only MLP structures with low overhead and equipped with explicit temporal interaction mechanisms. Extensive experiments on public datasets show that LSINet achieves both higher accuracy and better efficiency than advanced linear models and transformer models in TSF tasks. The code is available at the link https://github.com/Meteor-Stars/LSINet.

Figures (5)

• Figure 1: Forecasting error changes after retaining top $K$% higher self-attention scores (lower is set to zero) on the well-trained PatchTST model and test set of weather dataset.
• Figure 2: Illustration of repeated interaction patterns in self-attentions based on the test set of weather dataset and PatchTST model. Each batch consists of 64 test samples.
• Figure 3: LSINet and corresponding components.
• Figure 4: Efficiency analysis for best-performing linear models on Weather dataset (batch size 64) with training epoch time (a) and GPU memory usage (b).
• Figure 5: Learned shared sparse connection matrices by LSINet: (a)-(b) and learned self-attention matrices by PatchTST: (c)-(d) on the ETTm1 dataset. Both of them exhibit patterns of scattered points and blocks.