Context Matters: Leveraging Contextual Features for Time Series Forecasting

TL;DR

ContextFormer introduces a plug-and-play framework to fuse multimodal contextual metadata into pre-trained time-series forecasters via cross-attention. By coupling historical data with heterogeneous metadata (categorical, continuous, time-varying, and textual) and employing a fine-tuning scheme that freezes the base model, it guarantees at least as good performance as the original forecaster while achieving substantial accuracy gains across diverse domains. The approach is supported by information-theoretic motivation and empirical results showing improvements up to 30% in MSE/MAE over strong baselines, including across synthetic data and real-world datasets like Energy, Traffic, and Finance. This work offers a scalable path to context-aware forecasting without retraining foundational models from scratch, with potential extensions to additional modalities and meta-data forecasting.

Abstract

Time series forecasts are often influenced by exogenous contextual features in addition to their corresponding history. For example, in financial settings, it is hard to accurately predict a stock price without considering public sentiments and policy decisions in the form of news articles, tweets, etc. Though this is common knowledge, the current state-of-the-art (SOTA) forecasting models fail to incorporate such contextual information, owing to its heterogeneity and multimodal nature. To address this, we introduce ContextFormer, a novel plug-and-play method to surgically integrate multimodal contextual information into existing pre-trained forecasting models. ContextFormer effectively distills forecast-specific information from rich multimodal contexts, including categorical, continuous, time-varying, and even textual information, to significantly enhance the performance of existing base forecasters. ContextFormer outperforms SOTA forecasting models by up to 30% on a range of real-world datasets spanning energy, traffic, environmental, and financial domains.

Figures (11)

• Figure 1: Forecasting with context. A context-aware forecaster like ContextFormer can incorporate multimodal contextual information, such as daily news articles, online search trends, and market data, to enhance the accuracy of time-series forecasts.
• Figure 2: ContextFormer estimates the true effect of the explainable contextual factors on the forecast. Here, we show an example of a bitcoin price forecast using news articles and the historical price for the past four days. Note that the sharp decline in price is attributed to an ongoing market correction. Existing context-agnostic forecasting models treat this as a transient shock, leading to an overcorrection in price recovery. In contrast, our ContextFormer comprehends the underlying market dynamics, resulting in a more accurate and reliable forecast.
• Figure 3: Graphical models for forecasting. The figures represent the graphical models for the two forecasting approaches. In the context-aware model, the forecast $\hat{\bm{X}}_{\mathrm{future}}$ follows from both the history $\bm{X}_{\mathrm{hist}}$ and context $\bm{C}_{\mathrm{hist}}$, while for the context-agnostic model, $\hat{\bm{X}}_{\mathrm{future}}$ depends only on $\bm{X}_{\mathrm{hist}}$.
• Figure 4: Adding context improves the forecasting accuracy of an AR model. In this experiment, we vary the number of contextual features from 0 to 5 to demonstrate how the inclusion of these features reduces the MSE for a simple Autoregressive forecaster.
• Figure 5: ContextFormer Architecture. The architecture incorporates the multimodal contextual information in the form of metadata and temporal embeddings through a cross-attention-based method to improve the performance of an existing forecaster. During the fine-tuning phase of ContextFormer, the base model remains frozen, with only the final layer and newly added components being trained on paired contextual information.