Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reward-RAG: Enhancing RAG with Reward Driven Supervision

Thang Nguyen, Peter Chin, Yu-Wing Tai

TL;DR

Reward-RAG introduces a reward-driven supervision loop to improve retrieval in RAG by training a reward model with CriticGPT-generated feedback that mimics human preferences. The reward model guides domain-adaptive fine-tuning of the retrieval encoder, using synthetic $(q,d,r)$ data and hard-negative mining with an InfoNCE objective. Empirical results across open-domain and medical domain benchmarks show improved retrieval quality and generation relevance, with notable gains in PubMedQA and NQ, and strong ablations demonstrating the value of GPT-4o-based feedback and thoughtful prompting. This framework advances practical, scalable alignment of RAG systems to human preferences and domain-specific needs, reducing reliance on costly human annotations while enabling robust cross-domain performance.

Abstract

In this paper, we introduce Reward-RAG, a novel approach designed to enhance the Retrieval-Augmented Generation (RAG) model through Reward-Driven Supervision. Unlike previous RAG methodologies, which focus on training language models (LMs) to utilize external knowledge retrieved from external sources, our method adapts retrieval information to specific domains by employing CriticGPT to train a dedicated reward model. This reward model generates synthesized datasets for fine-tuning the RAG encoder, aligning its outputs more closely with human preferences. The versatility of our approach allows it to be effectively applied across various domains through domain-specific fine-tuning. We evaluate Reward-RAG on publicly available benchmarks from multiple domains, comparing it to state-of-the-art methods. Our experimental results demonstrate significant improvements in performance, highlighting the effectiveness of Reward-RAG in improving the relevance and quality of generated responses. These findings underscore the potential of integrating reward models with RAG to achieve superior outcomes in natural language generation tasks.

Reward-RAG: Enhancing RAG with Reward Driven Supervision

TL;DR

Reward-RAG introduces a reward-driven supervision loop to improve retrieval in RAG by training a reward model with CriticGPT-generated feedback that mimics human preferences. The reward model guides domain-adaptive fine-tuning of the retrieval encoder, using synthetic data and hard-negative mining with an InfoNCE objective. Empirical results across open-domain and medical domain benchmarks show improved retrieval quality and generation relevance, with notable gains in PubMedQA and NQ, and strong ablations demonstrating the value of GPT-4o-based feedback and thoughtful prompting. This framework advances practical, scalable alignment of RAG systems to human preferences and domain-specific needs, reducing reliance on costly human annotations while enabling robust cross-domain performance.

Abstract

In this paper, we introduce Reward-RAG, a novel approach designed to enhance the Retrieval-Augmented Generation (RAG) model through Reward-Driven Supervision. Unlike previous RAG methodologies, which focus on training language models (LMs) to utilize external knowledge retrieved from external sources, our method adapts retrieval information to specific domains by employing CriticGPT to train a dedicated reward model. This reward model generates synthesized datasets for fine-tuning the RAG encoder, aligning its outputs more closely with human preferences. The versatility of our approach allows it to be effectively applied across various domains through domain-specific fine-tuning. We evaluate Reward-RAG on publicly available benchmarks from multiple domains, comparing it to state-of-the-art methods. Our experimental results demonstrate significant improvements in performance, highlighting the effectiveness of Reward-RAG in improving the relevance and quality of generated responses. These findings underscore the potential of integrating reward models with RAG to achieve superior outcomes in natural language generation tasks.
Paper Structure (22 sections, 6 equations, 3 figures, 8 tables)

This paper contains 22 sections, 6 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Methodology
  4. Dense Retrieval in RAG
  5. User Preference Alignment using a Reward Model
  6. Reward Models
  7. Collecting LLMs' feedback
  8. Fine-tuning Retrieval Model
  9. Experiments
  10. Main Experiments
  11. Experiments Setup
  12. Results
  13. Domain Specific RAG Tasks
  14. Ablation Studies
  15. Compare Feedback From Different LLMs
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Overview of our Reward-RAG. Given a query and its knowledge database, a retrieval model is used to retrieve the top-$k$ relevant documents, which then are rated for the relevance by a CriticGPT. These $\langle query,\,document \rangle$ pairs and their CriticGPTs' feedback are used to train a reward model, which is used to fine-tune the RAG retrieval to better align with human preferences.
  • Figure 2: Overview of the reward method. We follow the design in wang2024interpretablepreferencesmultiobjectivereward to adapt an existing autoregressive model to be a reward model.
  • Figure 3: Confusion matrix between GPT-3.5's feedback and GPT-4o's feedback. Labels $0,1,$ and $2$ are corresponding to Not Relevant, Moderately Relevant, and Highly Relevant respectively.