Do Vision-Language Models See Visualizations Like Humans? Alignment in Chart Categorization

TL;DR

The paper investigates whether vision-language models align with human perception in parsing scientific visualizations, focusing on core perceptual attributes: purpose, dimensionality, and encoding. It employs a visual-only, zero-shot setup using 305 VisImageNavigator images from VIS30K and ground-truth VisType labels to evaluate 13 diverse vision-language systems via system prompts and a strict JSON output schema. Findings show robust identification of purpose and dimensionality but significant difficulty with fine-grained encodings, with model scale not guaranteeing better alignment and a tendency toward overconfident mislabeling. The work highlights a perceptual gap and argues for careful human supervision and benchmark-guided integration to advance trustworthy, human-centered visualization tools, while outlining next steps to broaden the evaluation and probe multimodal reasoning through textual cues.

Abstract

Vision-language models (VLMs) hold promise for enhancing visualization tools, but effective human-AI collaboration hinges on a shared perceptual understanding of visual content. Prior studies assessed VLM visualization literacy through interpretive tasks, revealing an over-reliance on textual cues rather than genuine visual analysis. Our study investigates a more foundational skill underpinning such literacy: the ability of VLMs to recognize a chart's core visual properties as humans do. We task 13 diverse VLMs with classifying scientific visualizations based solely on visual stimuli, according to three criteria: purpose (e.g., schematic, GUI, visualization), encoding (e.g., bar, point, node-link), and dimensionality (e.g., 2D, 3D). Using expert labels from the human-centric VisType typology as ground truth, we find that VLMs often identify purpose and dimensionality accurately but struggle with specific encoding types. Our preliminary results show that larger models do not always equate to superior performance and highlight the need for careful integration of VLMs in visualization tasks, with human supervision to ensure reliable outcomes.

Figures (1)

• Figure 1: F1-score heatmap comparing VLM classification to expert judgment for purpose, encoding, and dimensionality. Models (x-axis) and classification labels (y-axis) are sorted in descending order by their average F1-score. Darker colors mean higher agreement, yellow cells (F1-score $\approx 0$) indicate very low alignment, and white cells show where a model never produced the corresponding label at all. A none label indicates cases where experts intentionally omitted a category. VLMs generally perform better at identifying 2D dimensionality, simpler encodings like bar, line, point, and vis (visualization example) purpose. However, classifying specific encoding types remains challenging. The prevalence of yellow and white cells reveals that models tend to offer an incorrect label rather than abstain, a behavior underscored by their difficulty predicting the none category.