Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Foundation Model for Instruction-Conditioned In-Context Time Series Tasks

Anish Saha, Konstantin Shmakov

Abstract

In-context learning (ICL) allows a model to adapt at inference time by conditioning on examples rather than updating parameters. Existing time-series foundation models use implicit positional context, retrieval, or task-specific objectives, but rarely explicit instruction-conditioned demonstrations. We present a foundation model for instruction-conditioned in-context time-series tasks based on a quantile-regression T5 encoder-decoder. Historical examples and queries are encoded with a structured tokenization scheme that marks target series, covariates, context, and task-specific future information. A hierarchical Transformer with per-example encoding, example-level fusion, and cross-example attention conditions decoding on demonstration pairs, enabling forecasting and related tasks without task-specific fine-tuning. We train on large-scale real and synthetic time series using supervised forecasting plus self-supervised tasks, including imputation, reconstruction, classification, anomaly detection, and source demixing. This multi-task training learns a distribution over task mappings and improves adaptation to local structure at inference time. Across diverse datasets, frequencies, and horizons, our method outperforms strong foundation baselines on point and probabilistic forecasting benchmarks, including fev-bench and GIFT-Eval, while remaining competitive on classification and anomaly detection.

A Foundation Model for Instruction-Conditioned In-Context Time Series Tasks

Abstract

In-context learning (ICL) allows a model to adapt at inference time by conditioning on examples rather than updating parameters. Existing time-series foundation models use implicit positional context, retrieval, or task-specific objectives, but rarely explicit instruction-conditioned demonstrations. We present a foundation model for instruction-conditioned in-context time-series tasks based on a quantile-regression T5 encoder-decoder. Historical examples and queries are encoded with a structured tokenization scheme that marks target series, covariates, context, and task-specific future information. A hierarchical Transformer with per-example encoding, example-level fusion, and cross-example attention conditions decoding on demonstration pairs, enabling forecasting and related tasks without task-specific fine-tuning. We train on large-scale real and synthetic time series using supervised forecasting plus self-supervised tasks, including imputation, reconstruction, classification, anomaly detection, and source demixing. This multi-task training learns a distribution over task mappings and improves adaptation to local structure at inference time. Across diverse datasets, frequencies, and horizons, our method outperforms strong foundation baselines on point and probabilistic forecasting benchmarks, including fev-bench and GIFT-Eval, while remaining competitive on classification and anomaly detection.
Paper Structure (77 sections, 43 equations, 5 figures, 14 tables, 1 algorithm)

This paper contains 77 sections, 43 equations, 5 figures, 14 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Problem Formulation and ICL Setup
  5. Data Representation and Granularity
  6. Multivariate time series and covariates
  7. Examples and Queries
  8. In-context learning input
  9. Structured Tokenization and Input Construction
  10. Target and Covariate types
  11. Structured In-context learning input
  12. Amortized Meta-Learning Task Definition
  13. Model Architecture
  14. Base Time-Series Encoder $\phi$
  15. Hierarchical Attention Fusion
  16. ...and 62 more sections

Figures (5)

  • Figure 1: Results of the GIFT-Eval benchmark: Aggregated scores of the overall benchmark. Lower values are better. "Zero-shot Models" are not trained on this data. The In-domain Models are partly trained on the dataset (Overlap:: Moirai 2.0 19%, TimesFM-2.5 10%, TTM: 16%).
  • Figure 2: Results of the fev-bench benchmark: Aggregated scores of the overall benchmark. Lower values are better. "Zero-shot Models" are not trained on this data.
  • Figure 3: Overall and long-term performance on the GIFT-Eval benchmark. (Train-evaluation overlap: Moirai 2.0 19%, TimesFM-2.5 10%, TTM 16%)
  • Figure 4: Term length performance on the GIFT-Eval benchmark. (Train-evaluation overlap: Moirai 2.0 19%, TimesFM-2.5 10%, TTM 16%)
  • Figure 5: Result on univariate and multivariate inputs on the GIFT-Eval benchmark. (Train-evaluation overlap: Moirai 2.0 19%, TimesFM-2.5 10%, TTM 16%)