Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AI Foundation Model for Time Series with Innovations Representation

Lang Tong, Xinyi Wang

TL;DR

The paper addresses the challenge of building AI foundation models for time series in engineering where dynamics are governed by physical laws and causality rather than language. It introduces Time Series GPT (TS-GPT) grounded in Wiener-Kallianpur innovations representations, leveraging strong and weak innovations to extract an IID-uniform latent sequence that drives generation. A GAN-based pretraining regime for the innovations autoencoder and a Generative Probabilistic Forecasting (GPF) framework enable sampling from the conditional distribution $F_{t+T|t}$, yielding MMSE, MMAE, and quantile forecasts through ensembles. Empirical results on real-time NYISO LMP data show the weak innovations autoencoder (WIAE) approach provides superior CRPS and CPE performance relative to benchmarks, highlighting the practical impact for real-time energy market forecasting and control.

Abstract

This paper introduces an Artificial Intelligence (AI) foundation model for time series in engineering applications, where causal operations are required for real-time monitoring and control. Since engineering time series are governed by physical, rather than linguistic, laws, large-language-model-based AI foundation models may be ineffective or inefficient. Building on the classical innovations representation theory of Wiener, Kallianpur, and Rosenblatt, we propose Time Series GPT (TS-GPT) -- an innovations-representation-based Generative Pre-trained Transformer for engineering monitoring and control. As an example of foundation model adaptation, we consider Probabilistic Generative Forecasting, which produces future time series samples from conditional probability distributions given past realizations. We demonstrate the effectiveness of TS-GPT in forecasting real-time locational marginal prices using historical data from U.S. independent system operators.

AI Foundation Model for Time Series with Innovations Representation

TL;DR

The paper addresses the challenge of building AI foundation models for time series in engineering where dynamics are governed by physical laws and causality rather than language. It introduces Time Series GPT (TS-GPT) grounded in Wiener-Kallianpur innovations representations, leveraging strong and weak innovations to extract an IID-uniform latent sequence that drives generation. A GAN-based pretraining regime for the innovations autoencoder and a Generative Probabilistic Forecasting (GPF) framework enable sampling from the conditional distribution , yielding MMSE, MMAE, and quantile forecasts through ensembles. Empirical results on real-time NYISO LMP data show the weak innovations autoencoder (WIAE) approach provides superior CRPS and CPE performance relative to benchmarks, highlighting the practical impact for real-time energy market forecasting and control.

Abstract

This paper introduces an Artificial Intelligence (AI) foundation model for time series in engineering applications, where causal operations are required for real-time monitoring and control. Since engineering time series are governed by physical, rather than linguistic, laws, large-language-model-based AI foundation models may be ineffective or inefficient. Building on the classical innovations representation theory of Wiener, Kallianpur, and Rosenblatt, we propose Time Series GPT (TS-GPT) -- an innovations-representation-based Generative Pre-trained Transformer for engineering monitoring and control. As an example of foundation model adaptation, we consider Probabilistic Generative Forecasting, which produces future time series samples from conditional probability distributions given past realizations. We demonstrate the effectiveness of TS-GPT in forecasting real-time locational marginal prices using historical data from U.S. independent system operators.

Paper Structure

This paper contains 20 sections, 16 equations, 7 figures.

Table of Contents

  1. Introduction
  2. The Rise of AI Foundation Model
  3. Innovations Through the Lens of Modern Machine Learning
  4. The Wiener-Kallianpur Conjecture
  5. Weak Innovations Representation
  6. A Time Series GPT Foundation Model
  7. Pretraining Foundation Model with Innovations Representation
  8. Structural Convergence
  9. Probabilistic Forecasting GPT with Innovations Representation
  10. Generative Probabilistic Forecasting
  11. Structure of a Generative Probabilistic Forecaster
  12. From GPF to Point and Quantile Forecasting
  13. Performance Measure: CRPS, CPE and ACPE
  14. Continuous Ranked Probability Score (CRPS)
  15. Coverage Probability Error (CPE)
  16. ...and 5 more sections

Figures (7)

  • Figure 1: An AI Foundation Model.
  • Figure 2: Innovations autoencoder with causal encoder $G$ and decoder $H$.
  • Figure 3: A deep learning architecture of innovations representations.
  • Figure 4: Structure of Generative Probabilistic Forecaster (GPF).
  • Figure 5: Generative Probabilistic Forecaster (GPF).
  • ...and 2 more figures