Diagnosing Bottlenecks in Data Visualization Understanding by Vision-Language Models

TL;DR

FUGU introduces a synthetic benchmark of $3{,}968$ questions over $768$ scatter plots to diagnose bottlenecks in data-visualization understanding by vision-language models. Through activation patching and linear probes, the study shows that while vision encoders encode coordinates and spatial relations, the critical bottleneck lies in the handoff to the language model rather than encoding or arithmetic, with information becoming distributed across tokens in deeper layers. Providing ground-truth coordinates can improve simple tasks but does not generalize to ensemble reasoning, and fine-tuning yields only partial gains, underscoring architectural constraints in current VLMs. The work highlights the value of mechanistic interpretability for identifying failure modes in multimodal reasoning and points toward architectural and training strategies to enhance robust visualization understanding.

Abstract

Data visualizations are vital components of many scientific articles and news stories. Current vision-language models (VLMs) still struggle on basic data visualization understanding tasks, but the causes of failure remain unclear. Are VLM failures attributable to limitations in how visual information in the data visualization is encoded, how information is transferred between the vision and language modules, or how information is processed within the language module? We developed FUGU, a suite of data visualization understanding tasks, to precisely characterize potential sources of difficulty (e.g., extracting the position of data points, distances between them, and other summary statistics). We used FUGU to investigate three widely used VLMs. To diagnose the sources of errors produced by these models, we used activation patching and linear probes to trace information flow through models across a variety of prompting strategies. We found that some models fail to generate the coordinates of individual data points correctly, and these initial errors often lead to erroneous final responses. When these models are provided with the correct coordinates, performance improves substantially. Moreover, even when the model generates an incorrect response, the correct coordinates can be successfully read out from the latent representations in the vision encoder, suggesting that the source of these errors lies in the vision-language handoff. We further found that while providing correct coordinates helps with tasks involving one or a small number of data points, it generally worsens performance for tasks that require extracting statistical relationships across many data points. Fine-tuning models on FUGU also fails to yield ceiling performance. These findings point to architectural constraints in current VLMs that might pose significant challenges for reliable data visualization understanding.

Figures (13)

• Figure 1: A) The Fundamentals of Graph Understanding (FUGU) task suite and dataset. The top row (task suite) displays example prompts for each of our five tasks. The bottom row (sample size) shows example scatter plots from our dataset for each number of data points ($n$). FUGU pairs the five tasks with all applicable scatter plots, resulting in 3,968 unique <task, image> pairs. B) Visualization of the causal intervention method. We perform causal interventions on a counterfactual pair of scatter plots by passing each separately through the model with the same prompt, extracting embeddings from the vision encoder, and swapping the visual embeddings corresponding to a meaningful subset of image tokens (like a data point) from the source to the target run. If this successfully replaces model output behavior on target with the source prediction, then we have located a causally important part of the representation for that task.
• Figure 2: Model behavioral performance on FUGU. The $x$-axis represents the number of data points in each scatter plot, and the $y$-axis shows the accuracy of freely-generated VLM responses. Error bars show 95% CIs.
• Figure 3: Success rates for causal interventions on the vision encoders of LLaMA-3.2 (top row), LLaVA-OV (middle row), and InternVL (bottom row). Each individual plot shows success rates for causal interventions on a particular subset of image tokens at a particular vision encoder layer for each of the five tasks (indicated by the labels above the columns). The $x$-axis corresponds to vision encoder layers, while the $y$-axis shows the proportion of counterfactual pairs where the output answer on the target input was successfully swapped to the expected source output as a result of the intervention. The gray bars labeled " full layer" show baseline intervention accuracy for swapping the entire layer. The line labeled " dot" shows success rates when only intervening on tokens containing data points, while the line labeled " other" shows success rates for intervening on all non-dot tokens (including the CLS token).
• Figure 4: A) Examples of ensemble task stimuli and prompts.B) Providing ground truth points in the chain-of-thought improves performance for most tasks. The palest bars for each task show baseline behavioral accuracy when the model is allowed to freely generate (i.e. results from Figure \ref{['fig:behavioral_results']}). The blue "basic" bars show accuracy averaged across the basic FUGU tasks (see Figure \ref{['fig:task_gallery']}), while the orange "ensemble" bars show accuracy averaged across ensemble tasks (see Section \ref{['sec:ensemble']}). The middle bars show accuracy when the model is provided with its own generated coordinates for each data point in the scatter plot as part of its chain-of-thought, while the darkest bars show accuracy when models are given ground-truth coordinates. The performance gains offered by the ground-truth listing suggests that accurate coordinate extraction is a significant bottleneck.
• Figure 5: Probe test accuracy for position in vision vs. language layers. In each panel, the $x$-axis corresponds to model layer for the vision encoder (left, blue) and LLM (right, orange), while the $y$-axis shows mean probe test accuracy for position on that layer. Only vision layers that connect to the LLM (either via cross-attention in LLaMA-3.2 or as a direct input to the LLM context in LLaVA-OneVision and InternVL3) are shown. The asterisked LLM layers correspond to language blocks that receive visual inputs (again, either via cross-attention in LLaMA-3.2 or as input tokens for the other models).