Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contexts

TL;DR

This work tackles robustness in retrieval-augmented generation (RAG) for open-domain QA when retrieved contexts are noisy. It introduces Adaptive Contrastive Decoding (ACD), an entropy-based method that adaptively weights contextual influence during decoding by computing $\alpha_{ACD} = \dfrac{H(Y_t)}{H(Y_t) + H(Y_t^c)}$ and using $\hat{P}_{\theta}(Y_t|x,y_{<t}) = \text{softmax}(\mathbf{z}_t + \alpha_{ACD}(\mathbf{z}_t^c - \mathbf{z}_t))$. Empirical results on TriviaQA, Natural Questions, and PopQA show ACD outperforms baselines in the RAG setting and is particularly robust when the context is noisy or conflicting, with strong performance in Known-noisy scenarios. The findings demonstrate that adaptive weighting reduces disruption from noisy context while preserving the benefits of gold-context signals, advancing reliable knowledge-intensive QA with LLMs.

Abstract

When using large language models (LLMs) in knowledge-intensive tasks, such as open-domain question answering, external context can bridge the gap between external knowledge and the LLMs' parametric knowledge. Recent research has been developed to amplify contextual knowledge over the parametric knowledge of LLMs with contrastive decoding approaches. While these approaches could yield truthful responses when relevant context is provided, they are prone to vulnerabilities when faced with noisy contexts. We extend the scope of previous studies to encompass noisy contexts and propose adaptive contrastive decoding (ACD) to leverage contextual influence effectively. ACD demonstrates improvements in open-domain question answering tasks compared to baselines, especially in robustness by remaining undistracted by noisy contexts in retrieval-augmented generation.

Figures (4)

• Figure 1: An illustration of adaptive contrastive decoding (ACD). Entropy ($H$) changes depending on context relevance, affecting the adaptive weight ($\alpha_{\text{ACD}}$). Noisy context leads the model to incorrectly answer "Diede De Groot" when employing regular greedy decoding. ACD applies context-based adjustments, enabling the correct answer, "Sloane Stephens," despite the noise.
• Figure 2: EM accuracy of each method in Llama2-7B. EM of three datasets used are averaged for each subset, Unknown-gold and Known-noisy.
• Figure 3: EM accuracy on NQ-swap with contexts replacing the gold answer with a random entity span.
• Figure 4: EM across alpha values ranges from 0.0 to 1.0. The dashed line indicates EM score with $\alpha_{ACD}$.