When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs

TL;DR

The paper identifies Attention Distraction (AD) as a distinct failure mode in retrieval-augmented LVLMs, where retrieved context misallocates attention and hurts visual grounding even when knowledge is correct. It proposes MAD-RAG, a training-free intervention that decouples perception from context via a dual-question input and preserves image-conditioned evidence through attention mixing. Across OK-VQA, E-VQA, and InfoSeek, and multiple LVLM backbones, MAD-RAG delivers consistent improvements and recovers up to 74.68% of AD-related failures with negligible computational overhead. The work advances reliable knowledge-based visual reasoning by stabilizing cross-modal attention in long-context multimodal generation. It also lays groundwork for adaptive or prompt-based extensions to broader models and settings.

Abstract

While Retrieval-Augmented Generation (RAG) is one of the dominant paradigms for enhancing Large Vision-Language Models (LVLMs) on knowledge-based VQA tasks, recent work attributes RAG failures to insufficient attention towards the retrieved context, proposing to reduce the attention allocated to image tokens. In this work, we identify a distinct failure mode that previous study overlooked: Attention Distraction (AD). When the retrieved context is sufficient (highly relevant or including the correct answer), the retrieved text suppresses the visual attention globally, and the attention on image tokens shifts away from question-relevant regions. This leads to failures on questions the model could originally answer correctly without the retrieved text. To mitigate this issue, we propose MAD-RAG, a training-free intervention that decouples visual grounding from context integration through a dual-question formulation, combined with attention mixing to preserve image-conditioned evidence. Extensive experiments on OK-VQA, E-VQA, and InfoSeek demonstrate that MAD-RAG consistently outperforms existing baselines across different model families, yielding absolute gains of up to 4.76%, 9.20%, and 6.18% over the vanilla RAG baseline. Notably, MAD-RAG rectifies up to 74.68% of failure cases with negligible computational overhead.

Figures (10)

• Figure 1: Attention Distraction (AD) phenomenon introduced by RAG in LVLMs. We show the average image attention ratios over all heads and visualize the attention heatmaps (details in Sec. \ref{['sec:AD']}) at the decoding stage. We observe a previously overlooked failure mode, i.e., AD, in retrieval-augmented LVLMs, including (i) cross-modal distraction, and (ii) intra-image attention distraction.
• Figure 2: RAG-induced degradation and recovery by MAD-RAG on OK-VQA.Left: Sample-level performance dynamics of Vanilla RAG compared to the closed-book setting on OK-VQA. While RAG introduces gains (green), it causes some correct predictions to flip to incorrect (red), highlighting the issue of AD. Right: Accuracy comparison between Vanilla RAG and MAD-RAG. MAD-RAG consistently achieves higher performance by addressing the degradation issues shown on the left.
• Figure 3: Average image token attention ratio under closed-book and RAG settings across different VLM model families on OK-VQA dataset. Introducing retrieved textual context consistently reduces the image attention ratio, indicating a distraction of attention from visual inputs toward retrieved context. The attention is extracted at the last layer.
• Figure 4: Overview of Mitigating Attention Distraction in Retrieval-augmented Generation for LVLMs (MAD-RAG). The image is encoded into visual tokens and jointly processed with question and retrieved context tokens by LVLMs. At inference time, attention distributions are intervened by mixing attentions between two question tokens, mitigating context-induced attention distraction. Compared to vanilla RAG, the proposed intervention redirects attention toward question-relevant visual evidence and leads to the correct prediction.
• Figure 5: Improvement on different cases with LLaVA-1.5-7B. Rows and columns indicate whether the instance was correctly or incorrectly by the Closed-book and Vanilla RAG baselines, respectively. Cells report sample size (N), MAD-RAG's accuracy, and the change values ($\Delta$). The red box denotes the attention distraction cases (Closed-book $\checkmark$, RAG $\times$) recovered by MAD-RAG.