Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RPO: Retrieval Preference Optimization for Robust Retrieval-Augmented Generation

Shi-Qi Yan, Quan Liu, Zhen-Hua Ling

TL;DR

This work tackles knowledge conflicts in retrieval-augmented generation by integrating retrieval relevance into the training objective. The authors propose Retrieval Preference Optimization (RPO), a retrieval-aware alignment that combines generation and retrieval evaluation through a specialized reward model and a two-stage training pipeline (SFT followed by RPO). They demonstrate that RPO yields 4–10% accuracy gains across four benchmarks (PopQA, Natural Questions, TriviaQA, RGB) and offers robustness to low-quality retrieval and reduced inference overhead through Integrated-Eval. The approach provides a practical path toward more reliable RAG systems by explicitly modeling retrieval quality within the learning objective, with strong empirical evidence and ablation-supported insights on its components.

Abstract

While Retrieval-Augmented Generation (RAG) has exhibited promise in utilizing external knowledge, its generation process heavily depends on the quality and accuracy of the retrieved context. Large language models (LLMs) struggle to evaluate the correctness of non-parametric knowledge retrieved externally when it differs from internal memorization, leading to knowledge conflicts during response generation. To this end, we introduce the Retrieval Preference Optimization (RPO), a lightweight and effective alignment method to adaptively leverage multi-source knowledge based on retrieval relevance. An implicit representation of retrieval relevance is derived and incorporated into the reward model to integrate retrieval evaluation and response generation into a single model, solving the problem that previous methods necessitate the additional procedure to assess the retrieval quality. Notably, RPO is the only RAG-dedicated alignment approach that quantifies the awareness of retrieval relevance in training, overcoming mathematical obstacles. Experiments on four datasets demonstrate that RPO outperforms RAG by 4-10% in accuracy without any extra component, exhibiting its robust generalization.

RPO: Retrieval Preference Optimization for Robust Retrieval-Augmented Generation

TL;DR

This work tackles knowledge conflicts in retrieval-augmented generation by integrating retrieval relevance into the training objective. The authors propose Retrieval Preference Optimization (RPO), a retrieval-aware alignment that combines generation and retrieval evaluation through a specialized reward model and a two-stage training pipeline (SFT followed by RPO). They demonstrate that RPO yields 4–10% accuracy gains across four benchmarks (PopQA, Natural Questions, TriviaQA, RGB) and offers robustness to low-quality retrieval and reduced inference overhead through Integrated-Eval. The approach provides a practical path toward more reliable RAG systems by explicitly modeling retrieval quality within the learning objective, with strong empirical evidence and ablation-supported insights on its components.

Abstract

While Retrieval-Augmented Generation (RAG) has exhibited promise in utilizing external knowledge, its generation process heavily depends on the quality and accuracy of the retrieved context. Large language models (LLMs) struggle to evaluate the correctness of non-parametric knowledge retrieved externally when it differs from internal memorization, leading to knowledge conflicts during response generation. To this end, we introduce the Retrieval Preference Optimization (RPO), a lightweight and effective alignment method to adaptively leverage multi-source knowledge based on retrieval relevance. An implicit representation of retrieval relevance is derived and incorporated into the reward model to integrate retrieval evaluation and response generation into a single model, solving the problem that previous methods necessitate the additional procedure to assess the retrieval quality. Notably, RPO is the only RAG-dedicated alignment approach that quantifies the awareness of retrieval relevance in training, overcoming mathematical obstacles. Experiments on four datasets demonstrate that RPO outperforms RAG by 4-10% in accuracy without any extra component, exhibiting its robust generalization.
Paper Structure (35 sections, 19 equations, 3 figures, 4 tables)

This paper contains 35 sections, 19 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Adaptive RAG
  4. Model Alignment
  5. Task Definition
  6. RAG Formulation
  7. Knowledge Conflict
  8. Why DPO is Limited to Apply to RAG
  9. Firstly, the optimization objective of RLHF and DPO is inconsistent with the conflict-mitigating target in RAG.
  10. Secondly, the partition function within the reward model can not be canceled out.
  11. Thirdly, over-tendency towards non-parametric knowledge is still inevitable since parametric answers are fabricated for training.
  12. Methodology
  13. Theoretically Analysis
  14. Reward Model
  15. Length Normalization
  16. ...and 20 more sections

Figures (3)

  • Figure 1: The figure showcases the overview of RAG and three categories of adaptive RAG, including a) Pre-Eval, b) Post-Eval, and c) Integrated-Eval approaches. The estimated computational overhead of three categories is demonstrated as well, exhibiting the efficiency of our RPO in inference.
  • Figure 2: An overview of RPO at training. In phase 1, given a question and the retrieved documents, two answers $(y_p, y_n)$ are generated by the frozen language model $\pi$. After comparing with the golden answers, instances that involve knowledge conflict are filtered for supervised fine-tuning. In phase two, the fine-tuned LLM is prompted to generate a pair of answers again, and the instances with knowledge conflict are filtered as the training set of RPO.
  • Figure 3: Proportion of four clusters in PopQA and the corresponding accuracy scores on LLaMA2-7B.