Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RAG-RL: Advancing Retrieval-Augmented Generation via RL and Curriculum Learning

Jerry Huang, Siddarth Madala, Risham Sidhu, Cheng Niu, Hao Peng, Julia Hockenmaier, Tong Zhang

TL;DR

RAG-RL tackles the bottleneck in retrieval-augmented generation by training an answer generator with reinforcement learning to identify and cite relevant information from larger sets of retrieved passages. It uses curriculum learning to start with easier, gold-context-only examples and gradually handle distractors, yielding improved sample efficiency and generalization across three open-domain multi-hop QA datasets. The approach defines rule-based rewards for answer correctness, citation recall, and output formatting, and uses Group Relative Policy Optimization (GRPO) to optimize the policy. Across distractor-rich and ideal retrieval settings, RAG-RL achieves substantial gains in answer and citation accuracy, offering practical guidance on how training sample construction and reward design shape post-training success.

Abstract

Retrieval-augmented generation (RAG) systems rely on retrieval models for identifying relevant contexts and answer generation models for utilizing those contexts. However, retrievers exhibit imperfect recall and precision, limiting downstream performance. We introduce RAG-RL, an answer generation model trained not only to produce answers but also to identify and cite relevant information from larger sets of retrieved contexts, shifting some of the burden of identifying relevant documents from the retriever to the answer generator. Our approach uses curriculum learning, where the model is first trained on easier examples that include only relevant contexts. Our experiments show that these training samples enable models to acquire citation and reasoning skills with greater sample efficiency and generalizability, demonstrating strong model performance even as the number of irrelevant passages increases. We benchmark our methods on three open-domain multi-hop question answering datasets and report significant gains in answer and citation accuracy. Our experiments provide empirical insights into how easier training samples can give models stronger signals for learning specific skills (e.g., citation generation) and how different components of post-training (e.g., training set construction, rule-based rewards, training sample ordering, etc.) impact final model performance.

RAG-RL: Advancing Retrieval-Augmented Generation via RL and Curriculum Learning

TL;DR

RAG-RL tackles the bottleneck in retrieval-augmented generation by training an answer generator with reinforcement learning to identify and cite relevant information from larger sets of retrieved passages. It uses curriculum learning to start with easier, gold-context-only examples and gradually handle distractors, yielding improved sample efficiency and generalization across three open-domain multi-hop QA datasets. The approach defines rule-based rewards for answer correctness, citation recall, and output formatting, and uses Group Relative Policy Optimization (GRPO) to optimize the policy. Across distractor-rich and ideal retrieval settings, RAG-RL achieves substantial gains in answer and citation accuracy, offering practical guidance on how training sample construction and reward design shape post-training success.

Abstract

Retrieval-augmented generation (RAG) systems rely on retrieval models for identifying relevant contexts and answer generation models for utilizing those contexts. However, retrievers exhibit imperfect recall and precision, limiting downstream performance. We introduce RAG-RL, an answer generation model trained not only to produce answers but also to identify and cite relevant information from larger sets of retrieved contexts, shifting some of the burden of identifying relevant documents from the retriever to the answer generator. Our approach uses curriculum learning, where the model is first trained on easier examples that include only relevant contexts. Our experiments show that these training samples enable models to acquire citation and reasoning skills with greater sample efficiency and generalizability, demonstrating strong model performance even as the number of irrelevant passages increases. We benchmark our methods on three open-domain multi-hop question answering datasets and report significant gains in answer and citation accuracy. Our experiments provide empirical insights into how easier training samples can give models stronger signals for learning specific skills (e.g., citation generation) and how different components of post-training (e.g., training set construction, rule-based rewards, training sample ordering, etc.) impact final model performance.

Paper Structure

This paper contains 42 sections, 5 equations, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. RAG Systems
  4. Multi-Hop Question Answering
  5. Reasoning Language Models
  6. Curriculum Learning
  7. RAG-RL
  8. Reward Modeling
  9. Answer Rewards
  10. Citation Rewards
  11. Formatting Rewards
  12. Total Reward and Objective Function
  13. Curriculum Construction
  14. Synthetic Difficulty
  15. Accuracy-Based Difficulty
  16. ...and 27 more sections

Figures (5)

  • Figure 1: An example of a multi-hop reasoning chain taken from the MuSiQue dataset. RAG-RL generated the reasoning trace and final answer/citations observed in the green block.
  • Figure 2: Overview of two curriculum construction settings used during training. Linear denotes a curriculum that scales the difficulty level (the number of distractor passages) from 1 to $K$, while min-max denotes a curriculum that is split evenly between the easiest and the hardest problems.
  • Figure 3: Plots of validation answer and citation rewards during training for three curricula on the MuSiQue dataset.
  • Figure 4: System prompt used for all experiments.
  • Figure 5: User instructions used for all experiments.