ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models

TL;DR

ChartMuseum exposes a substantial gap between human and large vision-language models in chart understanding by separating visual and textual reasoning. It introduces a real-world, human-curated benchmark with 1,162 questions across 184 sources to probe complex visual and textual reasoning, complemented by a synthetic case study that isolates pure visual reasoning. Across 21 LVLMs, including open-source and proprietary models, humans achieve 93% accuracy while the best models reach only 63% (proprietary) and 38.5% (open-source), with visual reasoning lagging 35–55% behind textual reasoning. The work provides a visual-task taxonomy and error analysis that highlight specific limitations in current systems and establish ChartMuseum as a diagnostic platform to drive improvements in visual reasoning for chart understanding.

Abstract

Chart understanding presents a unique challenge for large vision-language models (LVLMs), as it requires the integration of sophisticated textual and visual reasoning capabilities. However, current LVLMs exhibit a notable imbalance between these skills, falling short on visual reasoning that is difficult to perform in text. We conduct a case study using a synthetic dataset solvable only through visual reasoning and show that model performance degrades significantly with increasing visual complexity, while human performance remains robust. We then introduce ChartMuseum, a new Chart Question Answering (QA) benchmark containing 1,162 expert-annotated questions spanning multiple reasoning types, curated from real-world charts across 184 sources, specifically built to evaluate complex visual and textual reasoning. Unlike prior chart understanding benchmarks -- where frontier models perform similarly and near saturation -- our benchmark exposes a substantial gap between model and human performance, while effectively differentiating model capabilities: although humans achieve 93% accuracy, the best-performing model Gemini-2.5-Pro attains only 63.0%, and the leading open-source LVLM Qwen2.5-VL-72B-Instruct achieves only 38.5%. Moreover, on questions requiring primarily visual reasoning, all models experience a 35%-55% performance drop from text-reasoning-heavy question performance. Lastly, our qualitative error analysis reveals specific categories of visual reasoning that are challenging for current LVLMs.

Figures (31)

• Figure 1: ChartMuseum contains a broad collection of charts with associated questions and answers, designed to test LVLMs at both textual and visual reasoning capabilities.
• Figure 2: Visual reasoning case study over histograms. We show the subplot and overlay setups on histograms in (a) and (c) with $n=4$. Results on Claude-3.7-Sonnet and humans over values of $n \in [4,9]$ are shown in (b) and (d). Complete questions can be found in Table \ref{['tab:visual-synthetic-questions']}.
• Figure 3: The composition of ChartMuseum: (a) depicts the distribution of chart types; (b) presents the sources from which these charts were collected; (c) displays the major question topics found in the dataset, tagged using Claude-3.7-Sonnet.
• Figure 4: The four categories of visual reasoning tasks that we identify and use for error categorization.
• Figure 5: We show a t-SNE visualization, comparing CLIP encodings radford2021learningtransferablevisualmodels from 1000 randomly sampled charts in ChartMuseum against those from several real-image based chart benchmarks.