Analyzing Deep Transformer Models for Time Series Forecasting via Manifold Learning

TL;DR

This study focuses on analyzing the geometric features of these latent data manifolds, including intrinsic dimension and principal curvatures, and reveals that deep transformer models exhibit similar geometric behavior across layers, and these geometric features are correlated with model performance.

Abstract

Transformer models have consistently achieved remarkable results in various domains such as natural language processing and computer vision. However, despite ongoing research efforts to better understand these models, the field still lacks a comprehensive understanding. This is particularly true for deep time series forecasting methods, where analysis and understanding work is relatively limited. Time series data, unlike image and text information, can be more challenging to interpret and analyze. To address this, we approach the problem from a manifold learning perspective, assuming that the latent representations of time series forecasting models lie next to a low-dimensional manifold. In our study, we focus on analyzing the geometric features of these latent data manifolds, including intrinsic dimension and principal curvatures. Our findings reveal that deep transformer models exhibit similar geometric behavior across layers, and these geometric features are correlated with model performance. Additionally, we observe that untrained models initially have different structures, but they rapidly converge during training. By leveraging our geometric analysis and differentiable tools, we can potentially design new and improved deep forecasting neural networks. This approach complements existing analysis studies and contributes to a better understanding of transformer models in the context of time series forecasting. Code is released at https://github.com/azencot-group/GATLM.

Figures (15)

• Figure 1: We study Transformer-based architectures wu2021autoformerzhou2022fedformer that include two encoders and one decoder, and an output linear layer. We sample geometric features in the output of sequence decomposition layers, depicted as solid blue blocks.
• Figure 2: Intrinsic dimension and mean absolute principal curvature along the layers of Autoformer and FEDformer on traffic dataset for multiple forecasting horizons. Top) intrinsic dimension. Bottom) mean absolute principal curvature. For each model, both ID and MAPC share a similar profile across different forecasting horizons.
• Figure 3: ID profiles across layers of Autoformer and FEDformer on electricity, traffic, weather and ETTm1 datasets for multiple forecasting horizons. Each panel includes ID profiles per dataset, for several horizons (left to right) and architectures (top to bottom).
• Figure 4: MAPC profiles across layers of Autoformer and FEDformer on electricity, traffic, weather and ETTm1 for multiple horizons. Each panel includes MAPC profiles per dataset, for several horizons (left to right) and architectures (top to bottom).
• Figure 5: MAPC is correlated with model performance. Each color represents a different dataset while the size of the dot is determined by the forecast horizon (longer horizon results in a larger dot). The test mean squared error is proportional to the MAPC on multiple datasets.