Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs

TL;DR

The paper addresses the robustness of Vision-Language Models (VLMs) to textual misinformation that conflicts with visual evidence. It introduces ConText-VQA, a dataset and multi-round benchmark framework that generates persuasive prompts targeting VLMs, and evaluates 11 diverse models, revealing an average accuracy drop of $48.2\%$ after the first round. Key contributions include a systematic misinformation-generation pipeline, a rigorous three-stage testing framework, and analyses of model behavior under sustained persuasion, including confidence shifts and cross-model differences between open-source and proprietary systems. A simple prompt-based alarm defense is proposed and shown to mitigate some vulnerability, underscoring the need for robustness-building approaches in practical, safety-critical multimodal AI deployments. The work highlights the risks of semantic manipulation in multimodal systems and motivates future research on architectural, training, and interaction-based defenses to ensure reliable perception and reasoning across modalities.

Abstract

Vision-Language Models (VLMs) have shown strong multimodal reasoning capabilities on Visual-Question-Answering (VQA) benchmarks. However, their robustness against textual misinformation remains under-explored. While existing research has studied the effect of misinformation in text-only domains, it is not clear how VLMs arbitrate between contradictory information from different modalities. To bridge the gap, we first propose the CONTEXT-VQA (i.e., Conflicting Text) dataset, consisting of image-question pairs together with systematically generated persuasive prompts that deliberately conflict with visual evidence. Then, a thorough evaluation framework is designed and executed to benchmark the susceptibility of various models to these conflicting multimodal inputs. Comprehensive experiments over 11 state-of-the-art VLMs reveal that these models are indeed vulnerable to misleading textual prompts, often overriding clear visual evidence in favor of the conflicting text, and show an average performance drop of over 48.2% after only one round of persuasive conversation. Our findings highlight a critical limitation in current VLMs and underscore the need for improved robustness against textual manipulation.

Figures (10)

• Figure 1: VLMs are susceptible to textual misinformation that conflicts with visual evidence, causing them to fail on questions they would otherwise answer correctly.
• Figure 2: Model performance measured as accuracy across different strategies over 4 rounds. We present the largest, most capable model from each family here to compare performance at the upper end of the scale.
• Figure 3: Performance comparison of the InternVL-3 model family. Here we show the average accuracy of each model across different strategies at each round.
• Figure 4: Confidence distribution for InternVL3-8B before/after persuasion, aggregated on all strategies.
• Figure 5: Confidence shift comparisons of all open-source models in our experiments. In each figure, left side is the confidence of the correct answer, right side is the confidence of the target choice at round 0 / 4. The last one is the average results.