ExoTST: Exogenous-Aware Temporal Sequence Transformer for Time Series Prediction

TL;DR

ExoTST addresses the challenge of incorporating current exogenous drivers with historical context in time-series forecasting. It introduces a two-branch exogenous encoder design and a cross-temporal fusion module within a Transformer-based encoder-decoder to exchange information across past and current exogenous modalities while predicting the endogenous series. The approach demonstrates improved accuracy on carbon flux GPP prediction and standard benchmarks, with robustness to missing and noisy exogenous inputs. This framework offers a scalable building block for long-horizon forecasting in domains where current exogenous signals are available.

Abstract

Accurate long-term predictions are the foundations for many machine learning applications and decision-making processes. Traditional time series approaches for prediction often focus on either autoregressive modeling, which relies solely on past observations of the target ``endogenous variables'', or forward modeling, which considers only current covariate drivers ``exogenous variables''. However, effectively integrating past endogenous and past exogenous with current exogenous variables remains a significant challenge. In this paper, we propose ExoTST, a novel transformer-based framework that effectively incorporates current exogenous variables alongside past context for improved time series prediction. To integrate exogenous information efficiently, ExoTST leverages the strengths of attention mechanisms and introduces a novel cross-temporal modality fusion module. This module enables the model to jointly learn from both past and current exogenous series, treating them as distinct modalities. By considering these series separately, ExoTST provides robustness and flexibility in handling data uncertainties that arise from the inherent distribution shift between historical and current exogenous variables. Extensive experiments on real-world carbon flux datasets and time series benchmarks demonstrate ExoTST's superior performance compared to state-of-the-art baselines, with improvements of up to 10\% in prediction accuracy. Moreover, ExoTST exhibits strong robustness against missing values and noise in exogenous drivers, maintaining consistent performance in real-world situations where these imperfections are common.

Figures (4)

• Figure 1: We illustrate four classes of problems in time series analysis, differentiated by their handling of exogenous drivers (white) and endogenous response (orange).
• Figure 2: ExoTST Architecture for time series prediction, featuring cross-temporal modality fusion and multi-head attention to integrate and predict based on past and future exogenous drivers.
• Figure 3: This plot visualizes daily GPP (Gross Primary Productivity) for ES-LJU with training, validation, and test data separated by color (red, green, blue).
• Figure 4: Visualization of the learned self-attention map alongside the endogenous look-back window for interpretation.