Patch-Level Tokenization with CNN Encoders and Attention for Improved Transformer Time-Series Forecasting

TL;DR

The paper tackles the challenge of Transformer-based time-series forecasting being highly sensitive to input representations. It introduces a two-stage architecture that first encodes fixed-length temporal patches with a CNN to produce patch-level tokens, refines them with token-level self-attention, and then uses a Transformer encoder to model inter-patch dependencies, yielding per-patch forecasts with equations such as $\hat{y}_{k+h}=w^\top h_k+b$. Evaluations on synthetic multivariate data with controlled dynamic/static factors show the method achieves competitive accuracy relative to strong baselines such as TCN and PatchTST, while offering increased learning stability through representation decoupling. The approach demonstrates that partitioning sequences into patches of length $P$, with $K=\big\lfloor T/P \big\rfloor$ patches, and forecasting via $\hat{y}_{k+h}=w^\top h_k+b$ provides a modular, extensible framework with potential applicability to real-world forecasting tasks.

Abstract

Transformer-based models have shown strong performance in time-series forecasting by leveraging self-attention to model long-range temporal dependencies. However, their effectiveness depends critically on the quality and structure of input representations derived from raw multivariate time-series data. This work proposes a two-stage forecasting framework that explicitly separates local temporal representation learning from global dependency modelling. In the first stage, a convolutional neural network (CNN) operates on fixed-length temporal patches to extract short-range temporal dynamics and non-linear feature interactions, producing compact patch-level token embeddings. Token-level self-attention is subsequently applied during representation learning to refine these embeddings by enabling interactions across temporal patches. In the second stage, a Transformer encoder processes the resulting token sequence to model inter-patch temporal dependencies and generate per-patch forecasts. Experiments conducted on synthetic multivariate time-series data with controlled static and dynamic factors demonstrate that the proposed patch-based tokenization strategy achieves competitive forecasting performance compared to convolutional and patch-based Transformer baselines. The results highlight the importance of structured temporal representations and show that decoupling local temporal encoding from global attention-based modelling yields more effective and stable time-series forecasting.

Figures (2)

• Figure 1: Overview of the proposed CNN--Transformer architecture for multivariate time-series forecasting.
• Figure 2: Comparison of forecasting performance across models using MSE and MAE.