Decoding Time Series with LLMs: A Multi-Agent Framework for Cross-Domain Annotation

TL;DR

The paper addresses the challenge of producing high-quality, cross-domain annotations for time-series data by introducing TESSA, a two-agent framework that jointly leverages general time-series knowledge and domain-specific terminology. It combines multi-modal feature extraction (time-series and text) with a hybrid adaptive feature selection strategy that includes offline LLM-based scoring and incremental RL-based updates, producing a general annotation $e_g$ via a general annotator and refining it into domain-specific annotations $e_t$ with a domain-specific component and reviewer. The approach is validated on synthetic and seven real-world datasets with multiple LLM backbones, showing superior annotation quality and demonstrable improvements in downstream forecasting, imputation, and domain-specific interpretation compared to baselines. The results suggest that cross-domain, multimodal annotation with domain-aware refinement can significantly enhance interpretability and performance in time-series analytics, while outlining practical considerations around data quality, annotation sources, and ethical use of LLMs.

Abstract

Time series data is ubiquitous across various domains, including manufacturing, finance, and healthcare. High-quality annotations are essential for effectively understanding time series and facilitating downstream tasks; however, obtaining such annotations is challenging, particularly in mission-critical domains. In this paper, we propose TESSA, a multi-agent system designed to automatically generate both general and domain-specific annotations for time series data. TESSA introduces two agents: a general annotation agent and a domain-specific annotation agent. The general agent captures common patterns and knowledge across multiple source domains, leveraging both time-series-wise and text-wise features to generate general annotations. Meanwhile, the domain-specific agent utilizes limited annotations from the target domain to learn domain-specific terminology and generate targeted annotations. Extensive experiments on multiple synthetic and real-world datasets demonstrate that TESSA effectively generates high-quality annotations, outperforming existing methods.

Figures (12)

• Figure 1: How to annotate time series across different domains?
• Figure 2: Overall framework of $\textnormal{TESSA}$. It consists of two main agents: a general annotation agent, which generates domain-independent annotations by selecting salient time-series and textual features, and a domain-specific annotation agent, which refines these annotations by incorporating domain-specific terminology.
• Figure 3: Comparison of offline vs. incremental feature selection. GPT-4o is the LLM backbone, with Environment as the target domain. (a) General annotation results; (b) Domain-specific annotation results.
• Figure 4: Overall framework of MM-TSFlib from Time-MMD liu2024time used in our multi-modal downstream tasks. MMTSFlib uses a model-agnostic multimodal integration framework that independently models time-series and textual annotations within an end-to-end training manner. MM-TSFlib slightly increases the number of trainable parameters, balancing effectiveness and efficiency.
• Figure 5: Selected time series data from Energy dataset for ablation studies on data contimination. This time series data has 30 data points. The corresponding generated annotations of $\textnormal{TESSA}$ and DirectLLMs are provided in Table \ref{['tab:olmo_ablation_studies']}.