Understanding Graphical Perception in Data Visualization through Zero-shot Prompting of Vision-Language Models

TL;DR

Evaluating the accuracy of zero-shot prompting of VLMs on graphical perception tasks with established human performance profiles reveals that VLMs perform similarly to humans under specific task and style combinations, suggesting that they have the potential to be used for modeling human performance.

Abstract

Vision Language Models (VLMs) have been successful at many chart comprehension tasks that require attending to both the images of charts and their accompanying textual descriptions. However, it is not well established how VLM performance profiles map to human-like behaviors. If VLMs can be shown to have human-like chart comprehension abilities, they can then be applied to a broader range of tasks, such as designing and evaluating visualizations for human readers. This paper lays the foundations for such applications by evaluating the accuracy of zero-shot prompting of VLMs on graphical perception tasks with established human performance profiles. Our findings reveal that VLMs perform similarly to humans under specific task and style combinations, suggesting that they have the potential to be used for modeling human performance. Additionally, variations to the input stimuli show that VLM accuracy is sensitive to stylistic changes such as fill color and chart contiguity, even when the underlying data and data mappings are the same.

Figures (4)

• Figure 1: Examples of the seven tasks in our study, adapted from heer2010crowdsourcing. For each visualization, the VLM was prompted to compare the two segments in blue and yellow (also labeled A and B, respectively).
• Figure 2: (a) Experiment 1 inverted colors and AB labels. (b) Stimulus variations in Experiment 2.
• Figure 3: Task variations in Experiment 3.
• Figure 4: Accuracy of VLMs on proportion judgments (probe 2).