ACFormer: Mitigating Non-linearity with Auto Convolutional Encoder for Time Series Forecasting

TL;DR

ACFormer addresses the non-linearity gap in time series forecasting by analyzing convolutional receptive fields and introducing an auto-convolutional encoder that blends linear projection efficiency with CNN-based feature extraction. The method introduces Shared Patch Compression, Temporal Gated Attention, and Independent Patch Expansion to capture fine-grained, variable-specific temporal patterns while maintaining computational efficiency. Empirical results on diverse real-world datasets show state-of-the-art performance and robustness to non-stationary signals, with notable gains on high-frequency components and superior efficiency relative to iTransformer. This work provides a principled framework for leveraging convolutional inductive biases in multivariate TSF, enabling practical deployment in large-scale, real-world settings.

Abstract

Time series forecasting (TSF) faces challenges in modeling complex intra-channel temporal dependencies and inter-channel correlations. Although recent research has highlighted the efficiency of linear architectures in capturing global trends, these models often struggle with non-linear signals. To address this gap, we conducted a systematic receptive field analysis of convolutional neural network (CNN) TSF models. We introduce the "individual receptive field" to uncover granular structural dependencies, revealing that convolutional layers act as feature extractors that mirror channel-wise attention while exhibiting superior robustness to non-linear fluctuations. Based on these insights, we propose ACFormer, an architecture designed to reconcile the efficiency of linear projections with the non-linear feature-extraction power of convolutions. ACFormer captures fine-grained information through a shared compression module, preserves temporal locality via gated attention, and reconstructs variable-specific temporal patterns using an independent patch expansion layer. Extensive experiments on multiple benchmark datasets demonstrate that ACFormer consistently achieves state-of-the-art performance, effectively mitigating the inherent drawbacks of linear models in capturing high-frequency components.

Figures (10)

• Figure 1: Residual extraction performance comparison between convolutional (Orange) and linear layers (Blue). The linear layers fail to reconstruct high-frequency components (marked by red circles), whereas convolutional layers succeed.
• Figure 2: Comparison between the (a) channel-wise attention from iTransformer and (b) our proposed variance attention from ModernTCN.
• Figure 3: The overall framework of ACFormer, illustrating the flow from Shared Patch Compression through Temporal Gated Attention to Independent Patch Expansion.
• Figure 4: Model efficiency(x-axis) and performance (y-axis) trade-off on the ECL dataset ($S=96, P=96$).
• Figure 5: Relative iteration time of ACFormer (Blue) and iTransformer(Orange) across the Weather, Solar, and Electricity datasets ($S=96, P=96$).