Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to Extract Rational Evidence via Reinforcement Learning for Retrieval-Augmented Generation

Xinping Zhao, Shouzheng Huang, Yan Zhong, Xinshuo Hu, Meishan Zhang, Baotian Hu, Min Zhang

TL;DR

This work tackles the noise and irrelevance problem inherent in retrieval-augmented generation by introducing EviOmni, a rational evidence extraction framework that first reasons about retrieved passages and then extracts concise, high-quality evidence. It integrates reasoning and extraction into a single on-policy reinforcement learning loop with verifiable rewards, aided by knowledge token masking to prevent leakage. Across five QA benchmarks, EviOmni improves answer accuracy, dramatically reduces evidence size via compression, and demonstrates robustness to retrieval noise, while also benefiting agentic RAG setups with lightweight, efficient evidence extraction. The approach offers practical gains in both open-domain and multi-hop QA, with limitations around inference latency suggesting future work on adaptive reasoning strategies. The key insight is that explicit reasoning before extraction yields more reliable clues, enabling tighter context and faster, more accurate generation.

Abstract

Retrieval-Augmented Generation (RAG) effectively improves the accuracy of Large Language Models (LLMs). However, retrieval noises significantly undermine the quality of LLMs' generation, necessitating the development of denoising mechanisms. Previous works extract evidence straightforwardly without deep thinking, which may risk filtering out key clues and struggle with generalization. To this end, we propose EviOmni, which learns to extract rational evidence via reasoning first and then extracting. Specifically, EviOmni integrates evidence reasoning and evidence extraction into one unified trajectory, followed by knowledge token masking to avoid information leakage, optimized via on-policy reinforcement learning with verifiable rewards in terms of answer, length, and format. Extensive experiments on five benchmark datasets show the superiority of EviOmni, which provides compact and high-quality evidence, enhances the accuracy of downstream tasks, and supports both traditional and agentic RAG systems.

Learning to Extract Rational Evidence via Reinforcement Learning for Retrieval-Augmented Generation

TL;DR

This work tackles the noise and irrelevance problem inherent in retrieval-augmented generation by introducing EviOmni, a rational evidence extraction framework that first reasons about retrieved passages and then extracts concise, high-quality evidence. It integrates reasoning and extraction into a single on-policy reinforcement learning loop with verifiable rewards, aided by knowledge token masking to prevent leakage. Across five QA benchmarks, EviOmni improves answer accuracy, dramatically reduces evidence size via compression, and demonstrates robustness to retrieval noise, while also benefiting agentic RAG setups with lightweight, efficient evidence extraction. The approach offers practical gains in both open-domain and multi-hop QA, with limitations around inference latency suggesting future work on adaptive reasoning strategies. The key insight is that explicit reasoning before extraction yields more reliable clues, enabling tighter context and faster, more accurate generation.

Abstract

Retrieval-Augmented Generation (RAG) effectively improves the accuracy of Large Language Models (LLMs). However, retrieval noises significantly undermine the quality of LLMs' generation, necessitating the development of denoising mechanisms. Previous works extract evidence straightforwardly without deep thinking, which may risk filtering out key clues and struggle with generalization. To this end, we propose EviOmni, which learns to extract rational evidence via reasoning first and then extracting. Specifically, EviOmni integrates evidence reasoning and evidence extraction into one unified trajectory, followed by knowledge token masking to avoid information leakage, optimized via on-policy reinforcement learning with verifiable rewards in terms of answer, length, and format. Extensive experiments on five benchmark datasets show the superiority of EviOmni, which provides compact and high-quality evidence, enhances the accuracy of downstream tasks, and supports both traditional and agentic RAG systems.

Paper Structure

This paper contains 31 sections, 11 equations, 8 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem Statement
  4. Empirical Study
  5. Methodology
  6. Obtaining Rational Evidence
  7. Modeling Verifiable Reward
  8. Online Policy Optimization
  9. Experiment
  10. Experimental Setup
  11. Main Comparison (RQ1)
  12. Training Dynamics (RQ2)
  13. Ablation Study (RQ3)
  14. Inference Efficiency (RQ4)
  15. Robustness Analysis (RQ5)
  16. ...and 16 more sections

Figures (8)

  • Figure 1: Motivating example, where key clues are marked in green: The key clue is omitted, leading to an incorrect answer; Guided by evidence reasoning, the key clue is extracted, leading to a correct answer.
  • Figure 2: The results w.r.t. AR on the in-distribution NQ as well as out-of-distribution HotoptQA datasets.
  • Figure 3: The overall framework of EviOmni and previous vanilla one. (a) Compared to vanilla evidence, rational evidence is more skilled in extracting key clues. (b) Existing works typically learn to extract vanilla evidence via SFT, where the output, i.e., oracle context containing key clues, is usually created by hand-crafted strategies. (c) Our EviOmni incentivizes rational evidence extraction capability in the extractor $\pi_\theta$ via RL with verifiable rewards.
  • Figure 4: Training dynamics w.r.t. answer reward and response length (including 'reason', 'extract', and 'answer').
  • Figure 5: Performance comparisons w.r.t. data noise.
  • ...and 3 more figures