Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChartDiff: A Large-Scale Benchmark for Comprehending Pairs of Charts

Rongtian Ye

Abstract

Charts are central to analytical reasoning, yet existing benchmarks for chart understanding focus almost exclusively on single-chart interpretation rather than comparative reasoning across multiple charts. To address this gap, we introduce ChartDiff, the first large-scale benchmark for cross-chart comparative summarization. ChartDiff consists of 8,541 chart pairs spanning diverse data sources, chart types, and visual styles, each annotated with LLM-generated and human-verified summaries describing differences in trends, fluctuations, and anomalies. Using ChartDiff, we evaluate general-purpose, chart-specialized, and pipeline-based models. Our results show that frontier general-purpose models achieve the highest GPT-based quality, while specialized and pipeline-based methods obtain higher ROUGE scores but lower human-aligned evaluation, revealing a clear mismatch between lexical overlap and actual summary quality. We further find that multi-series charts remain challenging across model families, whereas strong end-to-end models are relatively robust to differences in plotting libraries. Overall, our findings demonstrate that comparative chart reasoning remains a significant challenge for current vision-language models and position ChartDiff as a new benchmark for advancing research on multi-chart understanding.

ChartDiff: A Large-Scale Benchmark for Comprehending Pairs of Charts

Abstract

Charts are central to analytical reasoning, yet existing benchmarks for chart understanding focus almost exclusively on single-chart interpretation rather than comparative reasoning across multiple charts. To address this gap, we introduce ChartDiff, the first large-scale benchmark for cross-chart comparative summarization. ChartDiff consists of 8,541 chart pairs spanning diverse data sources, chart types, and visual styles, each annotated with LLM-generated and human-verified summaries describing differences in trends, fluctuations, and anomalies. Using ChartDiff, we evaluate general-purpose, chart-specialized, and pipeline-based models. Our results show that frontier general-purpose models achieve the highest GPT-based quality, while specialized and pipeline-based methods obtain higher ROUGE scores but lower human-aligned evaluation, revealing a clear mismatch between lexical overlap and actual summary quality. We further find that multi-series charts remain challenging across model families, whereas strong end-to-end models are relatively robust to differences in plotting libraries. Overall, our findings demonstrate that comparative chart reasoning remains a significant challenge for current vision-language models and position ChartDiff as a new benchmark for advancing research on multi-chart understanding.

Paper Structure

This paper contains 17 sections, 17 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Vision-Language Models for Chart Understanding
  4. Benchmarks for Chart Understanding
  5. Dataset Construction
  6. Raw Data Collection and Processing
  7. Chart Rendering
  8. Annotation Pipeline
  9. Dataset Curation Details
  10. Experiments
  11. Models
  12. Evaluation Metrics
  13. Implementation Details
  14. Analysis
  15. Conclusion
  16. ...and 2 more sections

Figures (17)

  • Figure 1: ChartDiff Dataset Illustration. The task requires comparing two charts and generating a concise description of their differences. More examples can be found in Appendix \ref{['sec:dataset_examples']}.
  • Figure 2: Fifty randomly selected chart pairs from the ChartDiff dataset.
  • Figure 3: An example pair of line charts.
  • Figure 4: An example pair of bar charts.
  • Figure 5: An example pair of horizontal bar charts.
  • ...and 12 more figures