TSLANet: Rethinking Transformers for Time Series Representation Learning

TL;DR

TSLANet reframes time series representation learning by substituting costly self-attention with an Adaptive Spectral Block that operates in the Fourier domain, augmented by an Interactive Convolution Block to capture multi-scale temporal patterns. A learnable adaptive thresholding mechanism jointly with global/local spectral filters suppresses high-frequency noise, while self-supervised masked autoencoding pretraining strengthens representations across diverse datasets. Empirical results across classification, forecasting, and anomaly detection reveal strong performance with substantially lower computational cost compared to Transformer-based models, supporting its potential as a robust foundation model for time series analysis. The work also provides comprehensive ablations and scalability insights, highlighting the value of frequency-domain processing and multi-kernel interactions for robust, efficient time series modeling.

Abstract

Time series data, characterized by its intrinsic long and short-range dependencies, poses a unique challenge across analytical applications. While Transformer-based models excel at capturing long-range dependencies, they face limitations in noise sensitivity, computational efficiency, and overfitting with smaller datasets. In response, we introduce a novel Time Series Lightweight Adaptive Network (TSLANet), as a universal convolutional model for diverse time series tasks. Specifically, we propose an Adaptive Spectral Block, harnessing Fourier analysis to enhance feature representation and to capture both long-term and short-term interactions while mitigating noise via adaptive thresholding. Additionally, we introduce an Interactive Convolution Block and leverage self-supervised learning to refine the capacity of TSLANet for decoding complex temporal patterns and improve its robustness on different datasets. Our comprehensive experiments demonstrate that TSLANet outperforms state-of-the-art models in various tasks spanning classification, forecasting, and anomaly detection, showcasing its resilience and adaptability across a spectrum of noise levels and data sizes. The code is available at https://github.com/emadeldeen24/TSLANet.

Figures (5)

• Figure 1: A comparison between CNN and Transformer-based architectures for classification and forecasting tasks. Classification results are the average over 10 UEA datasets Timesnet, while forecasting results are the average MSE results on lengths {96, 192, 336, 720}.
• Figure 2: The structure of our proposed TSLANet. The input time series is split into patches, and positional embeddings are added. Next, the output embeddings pass through TSLANet layers, where each layer consists of two main components. The first is the Adaptive Spectral Block, which leverages frequency domain representations for robust feature extraction and employs adaptive thresholding to mitigate noise. The second is the Interactive Convolution Block, which captures complex temporal patterns through convolutional operations.
• Figure 3: Effectiveness of the Adaptive Filter in noise reduction.
• Figure 4: A comparison between TSLANet vs. PatchTST in terms of accuracy with varying the number of layers in both for different data percentages from the uWaveGestureLibraryAll dataset.
• Figure 5: TSLANet vs. baselines in terms of the number of parameters and FLOPS count against the classification accuracy of the UEA Heartbeat dataset.