Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Long-Context Time Series Foundation Models

Nina Żukowska, Mononito Goswami, Michał Wiliński, Willa Potosnak, Artur Dubrawski

TL;DR

This study bridges the gap by cataloging and systematically comparing various context expansion techniques from both language and time series domains, and introducing a novel compressive memory mechanism to allow encoder-only TSFMs to effectively model intra-variate dependencies.

Abstract

Time series foundation models have shown impressive performance on a variety of tasks, across a wide range of domains, even in zero-shot settings. However, most of these models are designed to handle short univariate time series as an input. This limits their practical use, especially in domains such as healthcare with copious amounts of long and multivariate data with strong temporal and intra-variate dependencies. Our study bridges this gap by cataloging and systematically comparing various context expansion techniques from both language and time series domains, and introducing a novel compressive memory mechanism to allow encoder-only TSFMs to effectively model intra-variate dependencies. We demonstrate the benefits of our approach by imbuing MOMENT, a recent family of multi-task time series foundation models, with the multivariate context.

Towards Long-Context Time Series Foundation Models

TL;DR

This study bridges the gap by cataloging and systematically comparing various context expansion techniques from both language and time series domains, and introducing a novel compressive memory mechanism to allow encoder-only TSFMs to effectively model intra-variate dependencies.

Abstract

Time series foundation models have shown impressive performance on a variety of tasks, across a wide range of domains, even in zero-shot settings. However, most of these models are designed to handle short univariate time series as an input. This limits their practical use, especially in domains such as healthcare with copious amounts of long and multivariate data with strong temporal and intra-variate dependencies. Our study bridges this gap by cataloging and systematically comparing various context expansion techniques from both language and time series domains, and introducing a novel compressive memory mechanism to allow encoder-only TSFMs to effectively model intra-variate dependencies. We demonstrate the benefits of our approach by imbuing MOMENT, a recent family of multi-task time series foundation models, with the multivariate context.
Paper Structure (20 sections, 4 equations, 2 figures, 5 tables)

This paper contains 20 sections, 4 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Foundation Models
  4. Time Series Foundation Models
  5. Design Space of Multivariate Time Series Models
  6. Infini-Channel Mixer: Unlocking Multivariate Context using Compressive Memory
  7. Experiments and Results
  8. Supervised Long-horizon Forecasting Experiments.
  9. Pre-training Experiments.
  10. Discussion.
  11. Conclusion and Future Work
  12. Channel Mixing
  13. Concatenation Approach
  14. Infini-Channel Mixer + Static
  15. Training Experimental Setup
  16. ...and 5 more sections

Figures (2)

  • Figure 1: A design space of multivariate time series models. The proposed Infini-Channel Mixer is a homogeneous end-to-end channel mixing method.
  • Figure 2: Infini-Channel Mixer (ICM) uses a learned scalar $\beta$, to balance local information from dot product attention with global information from the compressive memory matrix which aggregates cross-channel information.