MMTS-BENCH: A Comprehensive Benchmark for Time Series Understanding and Reasoning

TL;DR

MMTS-BENCH introduces a hierarchical, multimodal benchmark for time-series understanding and reasoning, comprising 2,424 TSQA pairs across synthetic Base and real-world InWild, Match, Align subsets. It defines a five-dimensional taxonomy of tasks—Structural Awareness, Feature Analysis, Temporal Reasoning, Sequence Matching, and Cross-Modal Understanding—and uses both modular synthetic data and a progressive real-world QA pipeline to evaluate LLMs and TS-LLMs. Key findings show general-purpose LLMs outperform TS-LLMs in cross-domain generalization, local tasks lag global tasks, and that chain-of-thought reasoning and multimodal fusion substantially improve performance, while backbone scale is the dominant factor over encoder design. The work provides actionable guidance for future time-series foundation models, highlighting the value of CoT, multi-modality, and targeted prompt design to drive robust, interpretable, and generalizable temporal reasoning.

Abstract

Time series data are central to domains such as finance, healthcare, and cloud computing, yet existing benchmarks for evaluating various large language models (LLMs) on temporal tasks remain scattered and unsystematic. To bridge this gap, we introduce MMTS-BENCH, a comprehensive multimodal benchmark built upon a hierarchical taxonomy of time-series tasks, spanning structural awareness, feature analysis, temporal reasoning, sequence matching and cross-modal alignment. MMTS-BENCH comprises 2,424 time series question answering (TSQA) pairs across 4 subsets: Base, InWild, Match, and Align, generated through a progressive real-world QA framework and modular synthetic data construction. We conduct extensive evaluations on closed-source, open-source LLMs and existing time series adapted large language models (TS-LLMs), revealing that: (1) TS-LLMs significantly lag behind general-purpose LLMs in cross-domain generalization, (2) LLMs show weaknesses in local tasks compared to global tasks, (3) chain-of-thought (CoT) reasoning and multimodal integration substantially improve performance, and (4) the dominant factor in existing TS-LLMs remains the backbone network capability rather than the time series encoder design. MMTS-BENCH not only provides a rigorous evaluation framework but also offers clear directions for advancing LLMs toward robust, interpretable, and generalizable time-series reasoning.

Figures (11)

• Figure 1: MMTS-BENCH overview. Composition of the benchmark across subsets and source domains, with the number of QA instances in each component.
• Figure 2: Base Construction Pipeline. Synthetic time series with controllable characteristics are generated by concatenating and adding basic components of trend, seasonality, and noise. The plotting style of this figure is adapted from cai2024timeseriesexam.
• Figure 3: InWild Construction Pipeline. Flowchart illustrating the conversion of domain-specific time series via statistical analysis and multimodal input preparation for LLMs. It highlights the feedback loop between LLM generation and expert verification, showing how raw sequences from multiple domains (e.g., Economics, Transport, CloudOps, …) are enriched with features (e.g., trend strength, seasonal strength, entropy, …) and structured as inputs for automated QA generation.
• Figure 4: Match Construction Pipeline. Fragments are extracted from real-world time series, and candidate series with different similarity levels are retrieved using DTW. QA pairs are then formed with fixed templates, while transformations such as smoothing, extension, and reversal create four task paradigms of varying difficulty.
• Figure 5: Task-level accuracy of top-ranked LLMs on InWild (left: feature analysis; right: temporal reasoning).