TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Models

TL;DR

TSRBench addresses the lack of standardized, multi-domain evaluation for time-series reasoning in generalist systems by introducing a large-scale benchmark with 4125 problems across 14 domains, four capability dimensions (Perception, Reasoning, Prediction, Decision-Making), and four input modalities. The framework maps time-series to text for LLMs and to plots or embeddings for multimodal models, enabling uniform evaluation of both textual and visual representations. Key findings show that scaling improves performance on perception and reasoning tasks but not forecasting, and that textual and visual modalities are complementary yet not effectively fused by current models. The benchmark also provides principled data collection guidelines, ablation studies on visualization and reasoning, and insights into the challenges that must be addressed to build truly generalist time-series reasoning systems with practical impact.

Abstract

Time series data is ubiquitous in real-world scenarios and crucial for critical applications ranging from energy management to traffic control. Consequently, the ability to reason over time series is a fundamental skill for generalist models to solve practical problems. However, this dimension is notably absent from existing benchmarks of generalist models. To bridge this gap, we introduce TSRBench, a comprehensive multi-modal benchmark designed to stress-test the full spectrum of time series reasoning capabilities. TSRBench features: i) a diverse set of 4125 problems from 14 domains, and is categorized into 4 major dimensions: Perception, Reasoning, Prediction, and Decision-Making. ii) 15 tasks from the 4 dimensions evaluating essential reasoning capabilities (e.g., numerical reasoning). Through extensive experiments, we evaluated over 30 leading proprietary and open-source LLMs, VLMs, and TSLLMs within TSRBench. Our findings reveal that: i) scaling laws hold for perception and reasoning but break down for prediction; ii) strong reasoning does not guarantee accurate context-aware forecasting, indicating a decoupling between semantic understanding and numerical prediction; and iii) despite the complementary nature of textual and visual represenations of time series as inputs, current multimodal models fail to effectively fuse them for reciprocal performance gains. TSRBench provides a standardized evaluation platform that not only highlights existing challenges but also offers valuable insights to advance generalist models. Our code and dataset are available at https://tsrbench.github.io/.

Figures (28)

• Figure 1: Overview of TSRBench. TSRBench evaluates generalist models across four core capabilities: Perception, Reasoning, Prediction, and Decision-Making, each including multiple different tasks from real applications.
• Figure 2: Statistics of tasks in TSRBench.
• Figure 3: Overall accuracy and model sizes. Each plot illustrates the relationship between the log-scaled model size and the performance across all models. The left and right plots correspond to LLMs and VLMs, respectively.
• Figure 4: Spearman's rank correlation ($\rho$) between tasks. "(*)" marks correlations with p-values $\leq$ 0.05.
• Figure 5: Analysis of modality complementarity.Left: Comparison between textual and visual time series representations. Right: Ratio of model (T+V) answers identical to model (T) or model (V).