BEDTime: A Unified Benchmark for Automatically Describing Time Series

TL;DR

BEDTime introduces the first unified benchmark for describing univariate time series by evaluating three fundamental tasks—description recognition, description differentiation, and open description generation—across LLMs, VLMs, and TSLMs. By unifying four public datasets into a 10,164-pair corpus and applying consistent evaluation across image-based, text-based, and numeric modalities, the study reveals that vision-language models most effectively describe time series while language-only models struggle, and that all models are sensitive to realistic perturbations. The work provides a comprehensive evaluation framework with automatic NLI and human judgments and highlights the importance of visual representations and robust task designs for time-series understanding. It also offers actionable insights for future research, including developing stronger multi-modal time-series–language architectures and improving robustness under length, missingness, and noise, with code and data released to enable reproducibility.

Abstract

Recent works propose complex multi-modal models that handle both time series and language, ultimately claiming high performance on complex tasks like time series reasoning and cross-modal question-answering. However, they skip evaluations of simple and important foundational tasks, which complex models should reliably master. They also lack direct, head-to-head comparisons with other popular approaches. So we ask a simple question: Can recent models even produce generic visual descriptions of time series data? In response, we propose three new tasks, posing that successful multi-modal models should be able to recognize, differentiate, and generate language descriptions of time series. We then create BEDTime, the first benchmark dataset to assess models on each task, comprising four datasets reformatted for these tasks across multiple modalities. Using BEDTime, we evaluate 13 state-of-the-art models, and find that (1) surprisingly, dedicated time series foundation models severely underperform, despite being designed for similar tasks, (2) vision-language models are quite capable, (3) language-only methods perform worst, despite many lauding their potential, and (4) all approaches are clearly fragile to a range of realistic robustness tests, indicating avenues for future work.

Figures (5)

• Figure 1: Overview of the benchmark for automatic time series description (BEDTime), featuring head-to-head comparisons across three modalities. The benchmark includes two parts: a diverse collection of public datasets containing time series paired with textual descriptions of their visual properties (first row); and evaluation strategies across three tasks: description recognition, description differentiation, and open-ended generation given a time series (second row). The benchmark is compared head-to-head across three modalities (third row).
• Figure 2: Robustness of LLMs and TSLMs to sequence length, missingness, added gaussian noise and amplitude scaling. Recognition and differentiation accuracy decline for LLMs as time series grow longer or contain more missing values—particularly beyond 50. Also as the signals get noisier performance decline however models are relatively robust to amplitude scaling. DTW distance is used to pick dissimilar descriptions.
• Figure 3: Impact of Chain-of-Thought (CoT) prompting on language-only models' accuracy across four datasets and two tasks. CoT consistently improves both recognition and differentiation performance, with the largest gains observed on the differentiation task—especially for proprietary models. DTW distance is used to pick dissimilar descriptions.
• Figure 4: Accuracy of LLMs and VLMs on recognition and differentiation tasks across real-world (TRUCE-Stock) and synthetic (TRUCE-Synthetic, SUSHI, TaxoSynth) time series datasets. Negative samples for contrastive evaluation were generated using Dynamic Time Warping (DTW). The consistent performance gains of VLMs, especially on the differentiation task, highlight the importance of visual cues for robust time series analysis.
• Figure 5: Scaling for different time series lengths