Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Visualization Biases MLLM's Decision Making in Network Data Tasks

Timo Brand, Henry Förster, Stephen G. Kobourov, Jacob Miller

TL;DR

This study investigates how visualizations influence decision making in multi-modal LLMs when assessing whether a network contains a bridge. By comparing text-only adjacency representations with three visualization styles (AM, Circular, Spring) across two MLLMs (GPT and Qwen) and structured zero-shot prompts, the authors show that visualizations markedly bias judgments toward either a bridge or a 2-edge-connected structure, without improving overall accuracy. While visualization can elevate self-reported confidence, it tends to steer responses based on style rather than data, raising concerns about hallucinations in AI-assisted analysis. The work highlights the need for cautious visualization design in AI pipelines and motivates broader studies across tasks, layouts, and prompting schemes to understand and mitigate such biases.

Abstract

We evaluate how visualizations can influence the judgment of MLLMs about the presence or absence of bridges in a network. We show that the inclusion of visualization improves confidence over a structured text-based input that could theoretically be helpful for answering the question. On the other hand, we observe that standard visualization techniques create a strong bias towards accepting or refuting the presence of a bridge -- independently of whether or not a bridge actually exists in the network. While our results indicate that the inclusion of visualization techniques can effectively influence the MLLM's judgment without compromising its self-reported confidence, they also imply that practitioners must be careful of allowing users to include visualizations in generative AI applications so as to avoid undesired hallucinations.

Visualization Biases MLLM's Decision Making in Network Data Tasks

TL;DR

This study investigates how visualizations influence decision making in multi-modal LLMs when assessing whether a network contains a bridge. By comparing text-only adjacency representations with three visualization styles (AM, Circular, Spring) across two MLLMs (GPT and Qwen) and structured zero-shot prompts, the authors show that visualizations markedly bias judgments toward either a bridge or a 2-edge-connected structure, without improving overall accuracy. While visualization can elevate self-reported confidence, it tends to steer responses based on style rather than data, raising concerns about hallucinations in AI-assisted analysis. The work highlights the need for cautious visualization design in AI pipelines and motivates broader studies across tasks, layouts, and prompting schemes to understand and mitigate such biases.

Abstract

We evaluate how visualizations can influence the judgment of MLLMs about the presence or absence of bridges in a network. We show that the inclusion of visualization improves confidence over a structured text-based input that could theoretically be helpful for answering the question. On the other hand, we observe that standard visualization techniques create a strong bias towards accepting or refuting the presence of a bridge -- independently of whether or not a bridge actually exists in the network. While our results indicate that the inclusion of visualization techniques can effectively influence the MLLM's judgment without compromising its self-reported confidence, they also imply that practitioners must be careful of allowing users to include visualizations in generative AI applications so as to avoid undesired hallucinations.

Paper Structure

This paper contains 16 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Experimental Setup
  3. Generation of Stimuli
  4. Networks.
  5. Adjacency Information.
  6. Visualizations
  7. Trial MLLM Prompts
  8. Prompt Structure.
  9. System Message.
  10. Trial Instruction.
  11. Experimental Results
  12. Accuracy.
  13. Confidence.
  14. Discussion
  15. Conclusion
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Confidence intervals for the results of the experiments using the adjacency matrix-based configurations Novis, Novis$^*$ and AM. (left) shows the results for positive instances and (right) the negative instances. Note the inverted $x$-axis on the right.
  • Figure 2: (top) Confidence intervals for the results of the experiments using the baseline Novis and the visualization styles AM, Circular, and Spring. (bottom) Confidence intervals for the difference of means test between the baseline at (0.0), and the visualization styles. Intervals entirely to the right (left) of zero indicate significantly better (worse) performance than Novis. Note the inverted $x$-axis on the right.
  • Figure 3: Share of answers with a certain confidence value for (top) positive and (bottom) negative instances.