Why context matters in VQA and Reasoning: Semantic interventions for VLM input modalities

TL;DR

This paper systematically analyzes how image and text modalities contribute to Visual Language Model predictions in VQA and reasoning tasks using the SI-VQA dataset and ISI tool. It introduces seven modality configurations to probe complementary versus contradictory contexts, evaluating state-of-the-art VLMs (LLaVA variants and PaliGemma) across accuracy, reasoning quality, uncertainty, and attention attribution, with semantic entropy-based metrics like $SE(x)$ and the AUGRC measure for silent failures. Key findings show that complementary textual context improves accuracy and reasoning, while contradictory context degrades performance and confidence; image information dominates attention, and textual annotations offer limited benefit. The work also demonstrates that prompt engineering can modestly influence modality emphasis without reliably shifting attention, and highlights the problematic overconfidence of PaliGemma. Overall, SI-VQA and ISI provide a robust framework for rigorous analysis of modality integration and model reliability in multimodal predictions, with implications for dataset design and future VLM development.

Abstract

The various limitations of Generative AI, such as hallucinations and model failures, have made it crucial to understand the role of different modalities in Visual Language Model (VLM) predictions. Our work investigates how the integration of information from image and text modalities influences the performance and behavior of VLMs in visual question answering (VQA) and reasoning tasks. We measure this effect through answer accuracy, reasoning quality, model uncertainty, and modality relevance. We study the interplay between text and image modalities in different configurations where visual content is essential for solving the VQA task. Our contributions include (1) the Semantic Interventions (SI)-VQA dataset, (2) a benchmark study of various VLM architectures under different modality configurations, and (3) the Interactive Semantic Interventions (ISI) tool. The SI-VQA dataset serves as the foundation for the benchmark, while the ISI tool provides an interface to test and apply semantic interventions in image and text inputs, enabling more fine-grained analysis. Our results show that complementary information between modalities improves answer and reasoning quality, while contradictory information harms model performance and confidence. Image text annotations have minimal impact on accuracy and uncertainty, slightly increasing image relevance. Attention analysis confirms the dominant role of image inputs over text in VQA tasks. In this study, we evaluate state-of-the-art VLMs that allow us to extract attention coefficients for each modality. A key finding is PaliGemma's harmful overconfidence, which poses a higher risk of silent failures compared to the LLaVA models. This work sets the foundation for rigorous analysis of modality integration, supported by datasets specifically designed for this purpose.

Figures (28)

• Figure 1: The SI-VQA framework examines the influence of various modality configurations on answer accuracy and reasoning quality, model uncertainty, and attention attribution. Seven different configurations are tested, combining inputs such as the question (Q), image (I), annotated image (I$_{\text{A}}$), and either complementary (C$_{+}$) or contradictory context (C$_{-}$): (Q), (Q+I), (Q+I+C$_{+}$), (Q+I+C$_{-}$), (Q+I$_{\text{A}}$), (Q+I$_{\text{A}}$+C$_{+}$), and (Q+I$_{\text{A}}$+C$_{-}$). For each configuration, the VLM is assessed first on its answer and then on its reasoning. Furthermore, we establish prior assumptions regarding how each modality is expected to impact the model's behavior.
• Figure 2: Quality of VLM answers and reasoning in the seven modality configurations of question (Q), image (I), annotated image (I$_A$), complementary (C$_{+}$) and contradictory context (C$_{-}$). Answer accuracy is measured using the ground truth labels of our SI-VQA Dataset and reasoning quality is evaluated using the external scoring of GPT-4o as a judge. A significant drop of accuracy in the answer and reasoning is observed for all models when adding contradictory context, i.e., Q+I+C$_{-}$ and Q+I$_A$+C$_{-}$. Results for PaliGemma 3B are only displayed for answering (see \ref{['subsec:models']}).
• Figure 3: VLM uncertainty when generating answers and reasonings in the seven modality configurations of question (Q), image (I), annotated image (I$_A$), complementary (C$_{+}$) and contradictory context (C$_{-}$). Uncertainty is measured using the semantic entropy--the lower the entropy, the more confident the model. PaliGemma 3B shows extreme confidence overall in its answers. However, no reasoning results for PaliGemma are provided (see \ref{['subsec:models']}). C$_{-}$ negatively impacts the certainty of LLaVA models when generating answers.
• Figure 4: AUGRC evaluating the ability to detect silent failures through semantic entropy for each model and configuration (lower is better).
• Figure 5: Difference in attention attribution between the image and the question (solid line) and between the image and the context when present (dashed line). The computation of modality attention attribution is described in \ref{['apx:attention']}. The image (I) always gets the highest attention attribution compared to text modalities (Q, C$_{-}$, C$_{+}$). No reasoning results for PaliGemma are provided (see \ref{['subsec:models']}).