TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis

TL;DR

TOTEM advances generalist time-series analysis by learning discrete, fixed tokens from a large multi-domain corpus via self-supervised VQVAE training and applying a single forecasting backbone across imputation, anomaly detection, and forecasting. The approach emphasizes exclusive temporal tokenization and avoids domain-specific feature engineering, using RevIN normalization and a fixed codebook to produce tokens that can be stacked across sensors for multivariate data. Across nearly 500 experiments, TOTEM matches or exceeds state-of-the-art performance in both specialist and generalist settings and shows strong zero-shot generalization, often outperforming GPT2-based generalist baselines and PatchTOTEM baselines. The results suggest that discrete token representations can serve as a scalable, domain-agnostic foundation for time-series analysis, with practical impact on cross-domain research and deployment of time-series foundation models.

Abstract

This work studies the problem of time series analysis with generalist (or foundation) models, which are models trained across many data domains. Drawing inspiration from the widespread success of large language models, we consider the simple strategy of discretely tokenizing time series data drawn from a myriad of datasets via self-supervision, then using the fixed tokenization to solve a variety of tasks across many data domains. Canonically, time series models are either trained on a single dataset or built in a task-specific manner (e.g., a forecasting-only model), where many use patches of time as inputs to the model. As such, performant generalist, discrete representation time series models explored across many tasks are of value. Our method, TOkenized Time Series EMbeddings (TOTEM), produces such generalist time series models with minimal or no fine-tuning while exhibiting strong zero-shot performance. We evaluate TOTEM extensively over nearly 500 experiments on three commonly-studied time series tasks with real-world data: imputation (17 baselines, 12 datasets), anomaly detection (19 baselines, 25 datasets), and forecasting (14 baselines, 12 datasets). We conclude that TOTEM matches or outperforms existing state-of-the-art models in both the canonical specialist setting (i.e., training one model on one domain) as well as the generalist setting (i.e., training a single model on many domains), which demonstrates the efficacy of tokenization for general time series analysis. The open-source implementation is available here: https://github.com/SaberaTalukder/TOTEM; a video summary is available here: https://www.youtube.com/watch?v=OqrCpdb6MJk.

Figures (16)

• Figure 2: Left.Specialist models can tokenize along any of the $E$, $S$, or $T$ dimensions. Right.Generalist models can only tokenize along $T$, since the learned tokenization must apply to a diverse set of domains with any possible data dimensionality.
• Figure 3: TOTEM flattens the sensor and example dimensions and learns a discrete representation along the time dimension in a normalized space.
• Figure 4: The Forecaster Model. The forecaster takes in a tokenized version of normalized time series observations (obtained using TOTEM's encoder) and predicts a normalized time series over some specified horizon along with parameters that allow the model to unnormalize the prediction.
• Figure 5: Imputation Summary. In all categories TOTEM has SOTA AvgWins . In the specialist TOTEM has $52.1\%$AvgWins ; in generalist in domain TOTEM has $58.3\%$; in generalist zero shot TOTEM has $80.0\%$.
• Figure 6: Anomaly Detection Results. In all cases, TOTEM has SOTA AvgWins. Vs. specialists, TOTEM has $33.3\%$; vs. generalists in-domain, TOTEM has $80.0\%$; vs. generalists zero-shot, TOTEM has $73.3\%$.