Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TSAQA: Time Series Analysis Question And Answering Benchmark

Baoyu Jing, Sanhorn Chen, Lecheng Zheng, Boyu Liu, Zihao Li, Jiaru Zou, Tianxin Wei, Zhining Liu, Zhichen Zeng, Ruizhong Qiu, Xiao Lin, Yuchen Yan, Dongqi Fu, Jingchao Ni, Jingrui He, Hanghang Tong

TL;DR

TSAQA introduces a large-scale, unified benchmark for time series question answering that broadens evaluation beyond forecasting and anomaly detection to six advanced analytical tasks across 13 domains, using three question formats (TF, MC, PZ). It assembles 210k high-quality samples via hierarchical sampling, with standardized training/validation/testing splits and rigorous evaluation protocols, facilitating fair comparisons among commercial and open-source LLMs. Empirical results show that while instruction-tuned open models close some gaps, many tasks—especially Puzzle-based temporal reasoning (PZ) and data transformation involving Fourier transforms—remain challenging, revealing gaps in global temporal understanding and arithmetic reasoning. The paper also analyzes accuracy correlates with input length, domain, and task type, identifies a smoothness bias in PZ predictions, and validates a high-quality human-ground-truth pipeline, underscoring TSAQA’s usefulness for guiding future temporal reasoning research in LLMs.

Abstract

Time series data are integral to critical applications across domains such as finance, healthcare, transportation, and environmental science. While recent work has begun to explore multi-task time series question answering (QA), current benchmarks remain limited to forecasting and anomaly detection tasks. We introduce TSAQA, a novel unified benchmark designed to broaden task coverage and evaluate diverse temporal analysis capabilities. TSAQA integrates six diverse tasks under a single framework ranging from conventional analysis, including anomaly detection and classification, to advanced analysis, such as characterization, comparison, data transformation, and temporal relationship analysis. Spanning 210k samples across 13 domains, the dataset employs diverse formats, including true-or-false (TF), multiple-choice (MC), and a novel puzzling (PZ), to comprehensively assess time series analysis. Zero-shot evaluation demonstrates that these tasks are challenging for current Large Language Models (LLMs): the best-performing commercial LLM, Gemini-2.5-Flash, achieves an average score of only 65.08. Although instruction tuning boosts open-source performance: the best-performing open-source model, LLaMA-3.1-8B, shows significant room for improvement, highlighting the complexity of temporal analysis for LLMs.

TSAQA: Time Series Analysis Question And Answering Benchmark

TL;DR

TSAQA introduces a large-scale, unified benchmark for time series question answering that broadens evaluation beyond forecasting and anomaly detection to six advanced analytical tasks across 13 domains, using three question formats (TF, MC, PZ). It assembles 210k high-quality samples via hierarchical sampling, with standardized training/validation/testing splits and rigorous evaluation protocols, facilitating fair comparisons among commercial and open-source LLMs. Empirical results show that while instruction-tuned open models close some gaps, many tasks—especially Puzzle-based temporal reasoning (PZ) and data transformation involving Fourier transforms—remain challenging, revealing gaps in global temporal understanding and arithmetic reasoning. The paper also analyzes accuracy correlates with input length, domain, and task type, identifies a smoothness bias in PZ predictions, and validates a high-quality human-ground-truth pipeline, underscoring TSAQA’s usefulness for guiding future temporal reasoning research in LLMs.

Abstract

Time series data are integral to critical applications across domains such as finance, healthcare, transportation, and environmental science. While recent work has begun to explore multi-task time series question answering (QA), current benchmarks remain limited to forecasting and anomaly detection tasks. We introduce TSAQA, a novel unified benchmark designed to broaden task coverage and evaluate diverse temporal analysis capabilities. TSAQA integrates six diverse tasks under a single framework ranging from conventional analysis, including anomaly detection and classification, to advanced analysis, such as characterization, comparison, data transformation, and temporal relationship analysis. Spanning 210k samples across 13 domains, the dataset employs diverse formats, including true-or-false (TF), multiple-choice (MC), and a novel puzzling (PZ), to comprehensively assess time series analysis. Zero-shot evaluation demonstrates that these tasks are challenging for current Large Language Models (LLMs): the best-performing commercial LLM, Gemini-2.5-Flash, achieves an average score of only 65.08. Although instruction tuning boosts open-source performance: the best-performing open-source model, LLaMA-3.1-8B, shows significant room for improvement, highlighting the complexity of temporal analysis for LLMs.
Paper Structure (35 sections, 1 equation, 7 figures, 13 tables, 1 algorithm)

This paper contains 35 sections, 1 equation, 7 figures, 13 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. TSAQA Benchmark
  4. Task Formulation
  5. Data Collection
  6. Benchmark Construction
  7. Data Statistics
  8. Evaluation Protocol
  9. Experiments
  10. Main Results
  11. Analysis
  12. Accuracy Correlate Analysis
  13. Task Specific Analysis
  14. Case Study.
  15. Human Evaluation
  16. ...and 20 more sections

Figures (7)

  • Figure 1: Data distribution and tasks of TSAQA.
  • Figure 2: Histograms of time series, description and question lengths, and question type distribution.
  • Figure 3: Input lengths vs. Accuracy by Tasks among six models.
  • Figure 4: Input length vs. Accuracy by Question Types. CH, CP, DT, and TR denote Characterization, Comparison, Data Transformation, and Temporal Relationship. MC, TF, and PZ denote true-or-false, multiple-choice, and puzzling.
  • Figure 5: Topics vs. Accuracy of Comparison Task.
  • ...and 2 more figures