In-context Time Series Predictor

TL;DR

This work introduces the In-context Time Series Predictor (ICTSP), a Transformer-based approach that leverages in-context learning by treating forecasting tasks as input tokens formed from lookback–future pairs, enabling adaptive prediction without updating model parameters. ICTSP constructs context examples directly from ground-truth forecasting tasks, allowing the model to learn the best predictor for the target series via in-context prompts and robustly handle full-data, few-shot, and zero-shot settings. It further provides an adaptive model-reduction mechanism that can collapse to simpler linear or MLP predictors when temporal or inter-series dependencies are weak, and employs token retrieval to reduce computational costs. Across diverse multivariate TSF datasets, ICTSP demonstrates superior or competitive performance, especially in zero-shot transfer and few-shot regimes, highlighting its potential as a universal TSF framework with efficient in-context generalization capabilities.

Abstract

Recent Transformer-based large language models (LLMs) demonstrate in-context learning ability to perform various functions based solely on the provided context, without updating model parameters. To fully utilize the in-context capabilities in time series forecasting (TSF) problems, unlike previous Transformer-based or LLM-based time series forecasting methods, we reformulate "time series forecasting tasks" as input tokens by constructing a series of (lookback, future) pairs within the tokens. This method aligns more closely with the inherent in-context mechanisms, and is more parameter-efficient without the need of using pre-trained LLM parameters. Furthermore, it addresses issues such as overfitting in existing Transformer-based TSF models, consistently achieving better performance across full-data, few-shot, and zero-shot settings compared to previous architectures.

Figures (8)

• Figure 1: Overview of in-context TSF learning in our setup.
• Figure 2: Architecture and characteristic comparison among the three main TSF Transformer structures. Please note that, for simple illustration, the ICTSP part present a special case, where the sampling steps equal to $L_b+L_P$, creating non-overlapping context forecasting examples.
• Figure 3: Previous solutions of Temporal-wise Transformers' overfitting issue. From the ICL perspective, they are actually introducing more learnable temporal dependencies within token formulation.
• Figure 4: Comparison of the 3 architectures. The first 3 series of Multi and ETTm2 are visualized.
• Figure 5: Visualization of averaged attention maps of the 3 $\mathtt{TF}$ layers of ICTSP on Multi and ETTm2.