Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting

Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, Yan Liu

TL;DR

TEMPO introduces a foundation-model-inspired approach to time-series forecasting by integrating STL-based decomposition of trend, seasonality, and residual components with semi-soft prompts fed into a decoder-based GPT backbone. The method decomposes inputs, learns component-specific prompts, and uses LoRA for efficient adaptation, yielding per-component forecasts that are summed to produce the final prediction, with GAM/SHAP providing interpretability. In zero-shot evaluations across diverse benchmarks and in multimodal settings with contextual text, TEMPO achieves superior accuracy compared to both pre-trained LLM-based and conventional transformer baselines, demonstrating strong generalization to unseen domains. These results suggest that combining explicit temporal inductive biases with prompt-tuned generative models can establish a robust, adaptable foundation model framework for time-series forecasting, with practical implications for cross-domain and multimodal forecasting tasks.

Abstract

The past decade has witnessed significant advances in time series modeling with deep learning. While achieving state-of-the-art results, the best-performing architectures vary highly across applications and domains. Meanwhile, for natural language processing, the Generative Pre-trained Transformer (GPT) has demonstrated impressive performance via training one general-purpose model across various textual datasets. It is intriguing to explore whether GPT-type architectures can be effective for time series, capturing the intrinsic dynamic attributes and leading to significant accuracy improvements. In this paper, we propose a novel framework, TEMPO, that can effectively learn time series representations. We focus on utilizing two essential inductive biases of the time series task for pre-trained models: (i) decomposition of the complex interaction between trend, seasonal and residual components; and (ii) introducing the design of prompts to facilitate distribution adaptation in different types of time series. TEMPO expands the capability for dynamically modeling real-world temporal phenomena from data within diverse domains. Our experiments demonstrate the superior performance of TEMPO over state-of-the-art methods on zero shot setting for a number of time series benchmark datasets. This performance gain is observed not only in scenarios involving previously unseen datasets but also in scenarios with multi-modal inputs. This compelling finding highlights TEMPO's potential to constitute a foundational model-building framework.

TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting

TL;DR

TEMPO introduces a foundation-model-inspired approach to time-series forecasting by integrating STL-based decomposition of trend, seasonality, and residual components with semi-soft prompts fed into a decoder-based GPT backbone. The method decomposes inputs, learns component-specific prompts, and uses LoRA for efficient adaptation, yielding per-component forecasts that are summed to produce the final prediction, with GAM/SHAP providing interpretability. In zero-shot evaluations across diverse benchmarks and in multimodal settings with contextual text, TEMPO achieves superior accuracy compared to both pre-trained LLM-based and conventional transformer baselines, demonstrating strong generalization to unseen domains. These results suggest that combining explicit temporal inductive biases with prompt-tuned generative models can establish a robust, adaptable foundation model framework for time-series forecasting, with practical implications for cross-domain and multimodal forecasting tasks.

Abstract

The past decade has witnessed significant advances in time series modeling with deep learning. While achieving state-of-the-art results, the best-performing architectures vary highly across applications and domains. Meanwhile, for natural language processing, the Generative Pre-trained Transformer (GPT) has demonstrated impressive performance via training one general-purpose model across various textual datasets. It is intriguing to explore whether GPT-type architectures can be effective for time series, capturing the intrinsic dynamic attributes and leading to significant accuracy improvements. In this paper, we propose a novel framework, TEMPO, that can effectively learn time series representations. We focus on utilizing two essential inductive biases of the time series task for pre-trained models: (i) decomposition of the complex interaction between trend, seasonal and residual components; and (ii) introducing the design of prompts to facilitate distribution adaptation in different types of time series. TEMPO expands the capability for dynamically modeling real-world temporal phenomena from data within diverse domains. Our experiments demonstrate the superior performance of TEMPO over state-of-the-art methods on zero shot setting for a number of time series benchmark datasets. This performance gain is observed not only in scenarios involving previously unseen datasets but also in scenarios with multi-modal inputs. This compelling finding highlights TEMPO's potential to constitute a foundational model-building framework.
Paper Structure (46 sections, 3 theorems, 19 equations, 19 figures, 12 tables)

This paper contains 46 sections, 3 theorems, 19 equations, 19 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Problem Definition
  5. Time Series Input Representation
  6. Prompt Design
  7. Generative Pre-trained Transformer Architecture
  8. Experiments
  9. Zero Shot Long-term forecasting Results
  10. Short-term Forecasting with Contextual Information
  11. Analysis
  12. Ablation Study
  13. Interpreting Model Predictions
  14. Conclusion
  15. Showcases
  16. ...and 31 more sections

Key Result

Theorem 3.1

Suppose that we have time series signal $X = X_{Tt} + X_{St} + X_{Rt},t\in[t_1,t_n]$. Let $E=\{e_1,e_2,...,e_n\}$ denote a set of orthogonal bases. Let $E_S\subseteq E$ denote the subset of $E$ on which $X_{St}$ has non-zero eigenvalues and $E_T\subseteq E$ denote the subset of $E$ on which $X_{Tt}$

Figures (19)

  • Figure 1: The architecture of proposed TEMPO-GPT. The trend $X_T$, seasonal $X_S$ and residual $X_R$ components are treated as different semantic inductive biases to feed into the pre-trained transformer.
  • Figure 2: The SHAP values of decomposed components of TEMPO for ETTm1.
  • Figure 3: Ablation study on TEMPO.
  • Figure 3: Visualization of long-term forecasting results. Compared between our model TEMPO and GPT4TS on ETTh1 dataset
  • Figure 4: Visualization of long-term forecasting results. Compared between our model TEMPO and GPT4TS on ETTh2 dataset
  • ...and 14 more figures

Theorems & Definitions (5)

  • Theorem 3.1
  • Theorem G.1
  • Proof 1
  • Proposition G.2: Equivalence of time domain forecasting and frequency domain forecasting
  • Proof 2