Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

I'll believe it when I see it: Images increase misinformation sharing in Vision-Language Models

Alice Plebe, Timothy Douglas, Diana Riazi, R. Maria del Rio-Chanona

TL;DR

The paper investigates how images influence misinformation sharing by vision-language models, revealing that image presence increases resharing, especially for false content. It introduces a jailbreaking-inspired third-person prompting method and a PolitiFact-based multimodal dataset to study model behavior across architectures and persona conditions. Key findings show model- and persona-dependent amplification of false news with images, while certain models (e.g., Claude-3-Haiku) exhibit robustness to visual misinformation. The work underscores new risks in multimodal AI and advocates for targeted evaluation frameworks and mitigation strategies to curb persona- and image-driven misinformation spread.

Abstract

Large language models are increasingly integrated into news recommendation systems, raising concerns about their role in spreading misinformation. In humans, visual content is known to boost credibility and shareability of information, yet its effect on vision-language models (VLMs) remains unclear. We present the first study examining how images influence VLMs' propensity to reshare news content, whether this effect varies across model families, and how persona conditioning and content attributes modulate this behavior. To support this analysis, we introduce two methodological contributions: a jailbreaking-inspired prompting strategy that elicits resharing decisions from VLMs while simulating users with antisocial traits and political alignments; and a multimodal dataset of fact-checked political news from PolitiFact, paired with corresponding images and ground-truth veracity labels. Experiments across model families reveal that image presence increases resharing rates by 4.8% for true news and 15.0% for false news. Persona conditioning further modulates this effect: Dark Triad traits amplify resharing of false news, whereas Republican-aligned profiles exhibit reduced veracity sensitivity. Of all the tested models, only Claude-3-Haiku demonstrates robustness to visual misinformation. These findings highlight emerging risks in multimodal model behavior and motivate the development of tailored evaluation frameworks and mitigation strategies for personalized AI systems. Code and dataset are available at: https://github.com/3lis/misinfo_vlm

I'll believe it when I see it: Images increase misinformation sharing in Vision-Language Models

TL;DR

The paper investigates how images influence misinformation sharing by vision-language models, revealing that image presence increases resharing, especially for false content. It introduces a jailbreaking-inspired third-person prompting method and a PolitiFact-based multimodal dataset to study model behavior across architectures and persona conditions. Key findings show model- and persona-dependent amplification of false news with images, while certain models (e.g., Claude-3-Haiku) exhibit robustness to visual misinformation. The work underscores new risks in multimodal AI and advocates for targeted evaluation frameworks and mitigation strategies to curb persona- and image-driven misinformation spread.

Abstract

Large language models are increasingly integrated into news recommendation systems, raising concerns about their role in spreading misinformation. In humans, visual content is known to boost credibility and shareability of information, yet its effect on vision-language models (VLMs) remains unclear. We present the first study examining how images influence VLMs' propensity to reshare news content, whether this effect varies across model families, and how persona conditioning and content attributes modulate this behavior. To support this analysis, we introduce two methodological contributions: a jailbreaking-inspired prompting strategy that elicits resharing decisions from VLMs while simulating users with antisocial traits and political alignments; and a multimodal dataset of fact-checked political news from PolitiFact, paired with corresponding images and ground-truth veracity labels. Experiments across model families reveal that image presence increases resharing rates by 4.8% for true news and 15.0% for false news. Persona conditioning further modulates this effect: Dark Triad traits amplify resharing of false news, whereas Republican-aligned profiles exhibit reduced veracity sensitivity. Of all the tested models, only Claude-3-Haiku demonstrates robustness to visual misinformation. These findings highlight emerging risks in multimodal model behavior and motivate the development of tailored evaluation frameworks and mitigation strategies for personalized AI systems. Code and dataset are available at: https://github.com/3lis/misinfo_vlm
Paper Structure (42 sections, 2 equations, 7 figures, 8 tables)

This paper contains 42 sections, 2 equations, 7 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. LLMs and misinformation
  4. Persona conditioning in LLMs
  5. Misinformation and visual biases in humans
  6. Visual content and VLMs
  7. Methodology
  8. Inducing personalities and demographics
  9. Third-person prompting
  10. Self-generated personality prompting
  11. New multimodal news dataset
  12. Experimental setup
  13. Statistical analysis
  14. Results
  15. RQ1: Influence of visual information
  16. ...and 27 more sections

Figures (7)

  • Figure 1: Overview of the prompting pipeline used to elicit resharing decisions from VLMs.
  • Figure 2: Distribution of reshare ratings across the Likert scale for each model without persona conditioning. Results are grouped by prompt modality (image-plus-text vs. text-only) and news veracity (false vs. true). Invalid responses refer to cases where a non-ambiguous rating could not be extracted from the model's output.
  • Figure 3: (a) Proportion of yes responses under different personality traits (the Big Five, the Dark Triad, and no profile), shown separately for each model. (b) Proportion of yes responses under different demographic profiles (gender, age, race, and political affiliation), grouped by political affiliation and aggregated across all models.
  • Figure :
  • Figure :
  • ...and 2 more figures