Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do MLLMs See What We See? Analyzing Visualization Literacy Barriers in AI Systems

Mengli, Duan, Yuhe, Jiang, Matthew Varona, Carolina Nobre

TL;DR

This study investigates whether Multimodal Large Language Models read visualizations similarly to humans and why they fail. It introduces a barrier-centric framework, extending human visualization-literacy taxonomy to MLLMs, and analyzes 309 erroneous responses across four SOTA models using the reVLAT benchmark. The authors identify two machine-specific barrier classes—Visual Reasoning and Coherence barriers—alongside translation and perception barriers, and show that errors concentrate on color-intensive, segment-based charts, despite strong performance on simpler visuals. The work informs evaluation protocols and the design of reliable AI-driven visualization assistants, and suggests directions for scaling analyses, automating coding, and causal experimentation to better understand and mitigate these barriers.

Abstract

Multimodal Large Language Models (MLLMs) are increasingly used to interpret visualizations, yet little is known about why they fail. We present the first systematic analysis of barriers to visualization literacy in MLLMs. Using the regenerated Visualization Literacy Assessment Test (reVLAT) benchmark with synthetic data, we open-coded 309 erroneous responses from four state-of-the-art models with a barrier-centric strategy adapted from human visualization literacy research. Our analysis yields a taxonomy of MLLM failures, revealing two machine-specific barriers that extend prior human-participation frameworks. Results show that models perform well on simple charts but struggle with color-intensive, segment-based visualizations, often failing to form consistent comparative reasoning. Our findings inform future evaluation and design of reliable AI-driven visualization assistants.

Do MLLMs See What We See? Analyzing Visualization Literacy Barriers in AI Systems

TL;DR

This study investigates whether Multimodal Large Language Models read visualizations similarly to humans and why they fail. It introduces a barrier-centric framework, extending human visualization-literacy taxonomy to MLLMs, and analyzes 309 erroneous responses across four SOTA models using the reVLAT benchmark. The authors identify two machine-specific barrier classes—Visual Reasoning and Coherence barriers—alongside translation and perception barriers, and show that errors concentrate on color-intensive, segment-based charts, despite strong performance on simpler visuals. The work informs evaluation protocols and the design of reliable AI-driven visualization assistants, and suggests directions for scaling analyses, automating coding, and causal experimentation to better understand and mitigate these barriers.

Abstract

Multimodal Large Language Models (MLLMs) are increasingly used to interpret visualizations, yet little is known about why they fail. We present the first systematic analysis of barriers to visualization literacy in MLLMs. Using the regenerated Visualization Literacy Assessment Test (reVLAT) benchmark with synthetic data, we open-coded 309 erroneous responses from four state-of-the-art models with a barrier-centric strategy adapted from human visualization literacy research. Our analysis yields a taxonomy of MLLM failures, revealing two machine-specific barriers that extend prior human-participation frameworks. Results show that models perform well on simple charts but struggle with color-intensive, segment-based visualizations, often failing to form consistent comparative reasoning. Our findings inform future evaluation and design of reliable AI-driven visualization assistants.
Paper Structure (20 sections, 3 figures)

This paper contains 20 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visualization Literacy Assessment
  4. AI Systems Capability in Visualization Tasks
  5. Understanding MLLM Failure Modes
  6. Methodology
  7. Study Design
  8. Pilot Experiments
  9. Full Experiment
  10. Multi-level Analysis
  11. MLLM Visualization Literacy Barriers
  12. Translation Barriers
  13. Visual Perception Barriers
  14. Visual Reasoning Barriers
  15. Coherence Barriers
  16. ...and 5 more sections

Figures (3)

  • Figure 1: An example of an incorrect MLLM response on a bar chart task (Q4). Highlighted regions denote the model’s reported attention points and their misidentified maxima. The red horizontal line shows the threshold for correctly identifying countries with speeds above 15 Mbps.
  • Figure 2: Accuracy by chart type across 4 MLLMs. Cells are colored from red (lowest accuracy) to green (highest). This visualization highlights per-chart difficulty and model-specific weaknesses.
  • Figure 3: Error distribution across all chart types for Claude. Bars show the percentage of each mistake type per chart, grouped across taxonomy described in Section \ref{['sec:barrier']}.