TRACE: Grounding Time Series in Context for Multimodal Embedding and Retrieval

TL;DR

TRACE addresses the need to ground time-series data in domain-specific textual context to enable cross-modal retrieval and context-aware forecasting. It introduces a two-stage framework with Channel Identity Tokens and channel-biased attention, coupled with dual-level hard negative mining, yielding a powerful multimodal retriever and a strong standalone encoder. The method supports retrieval-augmented generation and demonstrates state-of-the-art retrieval performance and robust downstream forecasting and classification on weather and TimeMMD benchmarks. This cross-modal grounding enhances interpretability and generalization for time-series models, with practical impact in domains like weather forecasting and healthcare monitoring.

Abstract

The ubiquity of dynamic data in domains such as weather, healthcare, and energy underscores a growing need for effective interpretation and retrieval of time-series data. These data are inherently tied to domain-specific contexts, such as clinical notes or weather narratives, making cross-modal retrieval essential not only for downstream tasks but also for developing robust time-series foundation models by retrieval-augmented generation (RAG). Despite the increasing demand, time-series retrieval remains largely underexplored. Existing methods often lack semantic grounding, struggle to align heterogeneous modalities, and have limited capacity for handling multi-channel signals. To address this gap, we propose TRACE, a generic multimodal retriever that grounds time-series embeddings in aligned textual context. TRACE enables fine-grained channel-level alignment and employs hard negative mining to facilitate semantically meaningful retrieval. It supports flexible cross-modal retrieval modes, including Text-to-Timeseries and Timeseries-to-Text, effectively linking linguistic descriptions with complex temporal patterns. By retrieving semantically relevant pairs, TRACE enriches downstream models with informative context, leading to improved predictive accuracy and interpretability. Beyond a static retrieval engine, TRACE also serves as a powerful standalone encoder, with lightweight task-specific tuning that refines context-aware representations while maintaining strong cross-modal alignment. These representations achieve state-of-the-art performance on downstream forecasting and classification tasks. Extensive experiments across multiple domains highlight its dual utility, as both an effective encoder for downstream applications and a general-purpose retriever to enhance time-series models.

Figures (11)

• Figure 1: A Use Case of Text-to-Timeseries Retrieval
• Figure 2: Overview of TRACE. CIT stands for Channel Identity Tokens, which serve as a key bridge to connect two stages. MMTS denotes multimodal time series.
• Figure 3: Illustration of TRACE, which encodes multivariate time series using channel-biased attention and aligns token embeddings with its corresponding textual description (e.g., $\mathbf{z}_i$ and $\mathbf{z}_{\text{cxt}}$) through cross-attention and dual-level contrastive learning. $\mathbf{z}'_{\text{cxt}}$ indicates an in-batch hard negative sample.
• Figure 4: Ablation studies on patch length, positional embedding, and hidden dimension for (a) Reconstruction MSE, (b) Classification Accuracy (%), and (c) Average Forecasting MSE. (d) shows ablation studies on the number of retrieved instances ($R$) in the RAG pipeline.
• Figure 5: Retrieval performance under varying numbers of negative samples. The best is indicated by $\star$.