TiVaT: A Transformer with a Single Unified Mechanism for Capturing Asynchronous Dependencies in Multivariate Time Series Forecasting

TL;DR

TiVaT tackles multivariate time series forecasting by introducing a single unified JA attention module that jointly models temporal and inter-variate dependencies to capture asynchronous lead–lag dynamics. The architecture decomposes inputs into trend and seasonality, applies patch embedding, and uses JA blocks augmented with Distance-aware Time-Variate (DTV) sampling to form pattern-focused representations, refined via cross-attention. The approach achieves state-of-the-art results across eight real-world datasets, with ablations confirming the effectiveness of JA, DTV sampling, and the offset design, particularly in high-dimensional and non-stationary settings. This work advances practical MTS forecasting by offering a transparent, unified mechanism that robustly captures cross-axis interactions and asynchronous dependencies, enabling improved predictive performance in complex, real-world data.

Abstract

Multivariate time series (MTS) forecasting is vital across various domains but remains challenging due to the need to simultaneously model temporal and inter-variate dependencies. Existing channel-dependent models, where Transformer-based models dominate, process these dependencies separately, limiting their capacity to capture complex interactions such as lead-lag dynamics. To address this issue, we propose TiVaT (Time-variate Transformer), a novel architecture incorporating a single unified module, a Joint-Axis (JA) attention module, that concurrently processes temporal and variate modeling. The JA attention module dynamically selects relevant features to particularly capture asynchronous interactions. In addition, we introduce distance-aware time-variate sampling in the JA attention, a novel mechanism that extracts significant patterns through a learned 2D embedding space while reducing noise. Extensive experiments demonstrate TiVaT's overall performance across diverse datasets, particularly excelling in scenarios with intricate asynchronous dependencies.

Figures (9)

• Figure 1: Comparison of Transformer-based CD models' attention mechanisms, $L_N$ represents the patched time axis, $V$ denotes the variate axis, and $D$ indicates the dimensional space. The red box represents the feature serving as the query in attention, while the purple box represents the features serving as key-value pairs.
• Figure 2: Overview of TiVaT.
• Figure 3: Joint-Axis Attention Block.
• Figure 4: Qualitative analysis for DTV sampling. The black points represent the query feature, while other features are colored based on their cosine similarity to the query: red for high similarity and blue for low similarity. (a) and (b) represent 2D embedding spaces for the trend and seasonality components, $X^{Tr}$ and $X^{Se}$, of an input $X$, respectively.
• Figure 5: Visualization of grid maps illustrating a reference point and its relevant points extracted in JA attention module. (a) and (b) describe grid maps for different reference points along timestamps for specific variates in Weather dataset. The red box indicates the reference point and the yellow boxes represent the features strongly related to it across the variate and temporal dimensions.