Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

In-context Pre-trained Time-Series Foundation Models adapt to Unseen Tasks

Shangqing Xu, Harshavardhan Kamarthi, Haoxin Liu, B. Aditya Prakash

TL;DR

A framework, In-Context Time-series Pre-training (ICTP), restructures the original pre-training data to equip the backbone TSFM with In-Context Learning (ICL) capabilities, enabling adaptation to unseen tasks.

Abstract

Time-series foundation models (TSFMs) have demonstrated strong generalization capabilities across diverse datasets and tasks. However, existing foundation models are typically pre-trained to enhance performance on specific tasks and often struggle to generalize to unseen tasks without fine-tuning. To address this limitation, we propose augmenting TSFMs with In-Context Learning (ICL) capabilities, enabling them to perform test-time inference by dynamically adapting to input-output relationships provided within the context. Our framework, In-Context Time-series Pre-training (ICTP), restructures the original pre-training data to equip the backbone TSFM with ICL capabilities, enabling adaptation to unseen tasks. Experiments demonstrate that ICT improves the performance of state-of-the-art TSFMs by approximately 11.4% on unseen tasks without requiring fine-tuning.

In-context Pre-trained Time-Series Foundation Models adapt to Unseen Tasks

TL;DR

A framework, In-Context Time-series Pre-training (ICTP), restructures the original pre-training data to equip the backbone TSFM with In-Context Learning (ICL) capabilities, enabling adaptation to unseen tasks.

Abstract

Time-series foundation models (TSFMs) have demonstrated strong generalization capabilities across diverse datasets and tasks. However, existing foundation models are typically pre-trained to enhance performance on specific tasks and often struggle to generalize to unseen tasks without fine-tuning. To address this limitation, we propose augmenting TSFMs with In-Context Learning (ICL) capabilities, enabling them to perform test-time inference by dynamically adapting to input-output relationships provided within the context. Our framework, In-Context Time-series Pre-training (ICTP), restructures the original pre-training data to equip the backbone TSFM with ICL capabilities, enabling adaptation to unseen tasks. Experiments demonstrate that ICT improves the performance of state-of-the-art TSFMs by approximately 11.4% on unseen tasks without requiring fine-tuning.
Paper Structure (18 sections, 1 figure, 2 tables, 1 algorithm)

This paper contains 18 sections, 1 figure, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Time Series Foundation Models
  4. In-Context Learning
  5. Methodology
  6. Problem Definition
  7. Enabling Non-fine-tune Multi-task Adaptation for TSFMs
  8. In-Context Time-series Pre-training (ICTP)
  9. Experiments
  10. Settings
  11. Backbone Models
  12. Tasks Candidates
  13. Baselines
  14. Dataset
  15. Results
  16. ...and 3 more sections

Figures (1)

  • Figure 1: Similar to LLMs, existing TSFMs are pre-trained on single-task objectives using large-scale data. Yet, due to the inherent differences between time-series and language data, TSFMs are restricted to single-task adaptation. Instead, we propose In-Context Time-series Pre-training (ICTP), reformatting the original dataset into a multi-task context-following structure, enabling TSFMs to gain the in-context learning ability which finally leads to non-fine-tuning adaptation to unseen tasks.