OpenDecoder: Open Large Language Model Decoding to Incorporate Document Quality in RAG

TL;DR

OpenDecoder addresses robustness gaps in retrieval-augmented generation by injecting explicit document-quality indicators into the LLM decoding process. It constructs three signals—Retrieval relevance, Ranker relevance, and QPP score—from external information and integrates them through learnable attention-modulation parameters, aided by robustness training that exposes the model to noisy inputs. Across five QA benchmarks, OpenDecoder outperforms vanilla RAG and several strong baselines, with larger models yielding larger gains and normalization choices impacting performance. The method is flexible and plug-and-play with post-training and other external indicators, offering a practical path to more reliable, trust-worthy RAG systems.

Abstract

The development of large language models (LLMs) has achieved superior performance in a range of downstream tasks, including LLM-based retrieval-augmented generation (RAG). The quality of generated content heavily relies on the usefulness of the retrieved information and the capacity of LLMs' internal information processing mechanism to incorporate it in answer generation. It is generally assumed that the retrieved information is relevant to the question. However, the retrieved information may have a variable degree of relevance and usefulness, depending on the question and the document collection. It is important to take into account the relevance of the retrieved information in answer generation. In this paper, we propose OpenDecoder, a new approach that leverages explicit evaluation of the retrieved information as quality indicator features for generation. We aim to build a RAG model that is more robust to varying levels of noisy context. Three types of explicit evaluation information are considered: relevance score, ranking score, and QPP (query performance prediction) score. The experimental results on five benchmark datasets demonstrate the effectiveness and better robustness of OpenDecoder by outperforming various baseline methods. Importantly, this paradigm is flexible to be integrated with the post-training of LLMs for any purposes and incorporated with any type of external indicators.

Figures (8)

• Figure 1: Comparison between the existing decoding LLMs that use their default probability distribution and our proposed approach that modifies the distribution by leveraging external explicit relevance signals.
• Figure 2: The framework of OpenDecoder, including Searching External Information with top-k retrieved documents, Indicators Construction based on the retrieved documents with various types of quality scores, teaching the model to leverage external explicit quality indicators for the Decoding Computation of LLM by modulating internal attention score computation and applying Robust Training, and finally obtaining the reshaped token probability distribution during content generation.
• Figure 3: Performance of aggregating various scores as guidance features across different evaluation settings and datasets.
• Figure 4: Performance of normalizing scores features with various approaches across different evaluation settings and datasets.
• Figure 5: The performance of using various top-k retrieved documents in the normal evaluation setting.