CaTS-Bench: Can Language Models Describe Time Series?

TL;DR

CaTS-Bench addresses the gap in time-series captioning by integrating numeric reasoning, contextual metadata, and visual plots into a context-aware benchmark. It introduces a 1746 HR gold standard plus a ~20k synthetic-caption corpus, validated for factuality and diversity, and a 910-question diagnostic suite to probe numeric precision and multimodal grounding. Experiments show proprietary VLMs provide strong baselines, but struggle with precise numeric descriptions; finetuning open-source models on synthetic data yields substantial gains and reveals limitations in visual grounding. The benchmark establishes a rigorous, scalable platform for evaluating grounded multimodal generation in numeric domains and guides future work on better numeric-visual integration.

Abstract

Time series captioning, the task of describing time series in natural language, requires numeric and temporal reasoning, trend interpretation, and contextual understanding. Existing benchmarks, however, often rely on fully synthetic or generic captions, and typically neglect metadata and visual representations. We introduce \textbf{CaTS-Bench}, a comprehensive benchmark for \textbf{C}ontext-\textbf{a}ware \textbf{T}ime \textbf{S}eries reasoning across $11$ diverse domains, centered on a gold-standard evaluation set of $1746$ human-rewritten captions that measure how effectively models translate numeric trends into immediately interpretable narratives. To address the scarcity of human-annotated data, we also propose a scalable pipeline for generating high-fidelity synthetic captions, the quality of which we validate. We evaluate leading Vision-Language Models on our benchmark, revealing that even proprietary models struggle to capture numeric nuances in temporal descriptions, while finetuning open-source models on synthetic data yields substantial performance gains. Finally, release a diagnostic suite of $910$ multiple-choice questions and tailored numeric metrics to gauge time-series-specific reasoning capabilities, establishing CaTS-Bench as a reliable foundation for grounded, multimodal language generation in numeric domains.

Figures (33)

• Figure 1: Overview of CaTS-Bench. It features $11$ domains organized into $7$ macro domains, provides human-rewritten and synthetic data, and formulates six challenging tasks, with time series captioning as the primary one.
• Figure 2: Overview of CaTS-Bench's dual-stream data generation pipeline. A time series window is cropped, metadata is attached, and a human or LLM writes a reference caption. See Appendix \ref{['example samples']} for examples and Appendix \ref{['human-verify']} for the quality verification protocol.
• Figure 4: MCQ accuracy (%) for exact temporal value lookup at different query positions. Short_Mid2 denotes sequences of length 6; other columns use long time series with queries at different temporal offsets. Results are averaged over 75 questions per temporal bucket.
• Figure 5: Performance change when adding visual input. Blue indicates improvement; red indicates degradation.
• Figure 6: Performance deltas on statistical inference with and without annotations. Blue indicates improvement; red degradation.