Relevance-aware Multi-context Contrastive Decoding for Retrieval-augmented Visual Question Answering

TL;DR

RMCD introduces a training-free decoding method for retrieval-augmented visual question answering that jointly leverages multiple retrieved contexts through relevance-based weighting and contrastive aggregation. By computing context weights from retrieval scores and combining per-context logits with deflection for low-relevance contexts, RMCD amplifies useful evidence while suppressing noise, further enhanced by an ensemble plausibility constraint to restrict sampling to plausible vocabularies. The approach yields consistent improvements across InfoSeek, Encyclopedic-VQA, and OK-VQA benchmarks with multiple LVLM backbones, and demonstrates robustness to retrieval quality while offering reduced decoding latency. This work enables a practical, model-agnostic enhancement to RAG-based VQA without additional training, accelerating deployment in knowledge-intensive multimodal tasks.

Abstract

Despite the remarkable capabilities of Large Vision Language Models (LVLMs), they still lack detailed knowledge about specific entities. Retrieval-augmented Generation (RAG) is a widely adopted solution that enhances LVLMs by providing additional contexts from an external Knowledge Base. However, we observe that previous decoding methods for RAG are sub-optimal as they fail to sufficiently leverage multiple relevant contexts and suppress the negative effects of irrelevant contexts. To this end, we propose Relevance-aware Multi-context Contrastive Decoding (RMCD), a novel decoding method for RAG. RMCD outputs a final prediction by combining outputs predicted with each context, where each output is weighted based on its relevance to the question. By doing so, RMCD effectively aggregates useful information from multiple relevant contexts while also counteracting the negative effects of irrelevant ones. Experiments show that RMCD consistently outperforms other decoding methods across multiple LVLMs, achieving the best performance on three knowledge-intensive visual question-answering benchmarks. Also, RMCD can be simply applied by replacing the decoding method of LVLMs without additional training. Analyses also show that RMCD is robust to the retrieval results, consistently performing the best across the weakest to the strongest retrieval results. Code is available at https://github.com/mlvlab/RMCD.

Figures (7)

• Figure 1: Motivation. RMCD generates better responses without additional training by contrasting outputs based on relevance of contexts, thereby effectively leveraging multiple relevant contexts and counteracting irrelevant contexts.
• Figure 2: Relevance-aware Multi-context Contrastive Decoding (RMCD). An LVLM $\mathcal{M}_{\theta}$ outputs multiple predictions $q(y_i| c_j)$ provided contexts $c_{j} \in \mathcal{C}^r$ retrieved from a knowledge base. Using the retrieval score $s_{c_j}$ of a context, a context weight $\alpha_j$ is obtained. Then, RMCD combines multiple predictions by contrasting multiple contexts, where each prediction is weighted by $\alpha_j$. Thereby, outputs with high $\alpha_j$ are effectively reflected while outputs with low $\alpha_j$ are deflected as illustrated in the figure.
• Figure 3: Accuracy by the number of contexts $n$ used.
• Figure 4: Retrieval Recall@$k$ ($k=1, 3, 5$) by the size of KBs.
• Figure 5: Accuracy by the size of KBs. Retrieval gets more difficult in larger KBs.