Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation

TL;DR

This work addresses unfaithful intermediate reasoning in reinforcement-learning–driven retrieval-augmented generation by proposing a formal evaluation framework for faithfulness in agentic search and introducing VERITAS, a training paradigm that injects fine-grained faithfulness rewards into the RL loop. The framework defines three faithfulness metrics—Information-Think, Think-Search, and Think-Answer—and demonstrates that traditional outcome-based training can yield faithful-looking answers with unfaithful reasoning. VERITAS combines a multi-component reward with a distilled reward model to enable scalable supervision, and empirical results show substantial improvements in Information-Think faithfulness (about a 15.3% gain) and modest gains in Think-Answer faithfulness (about 3.2%), while maintaining competitive task performance across seven QA benchmarks. The approach offers a path toward more reliable, transparent AI agents in retrieval-augmented tasks by aligning intermediate reasoning with evidence gathered during search.

Abstract

Inspired by the success of reinforcement learning (RL) in Large Language Model (LLM) training for domains like math and code, recent works have begun exploring how to train LLMs to use search engines more effectively as tools for retrieval-augmented generation. Although these methods achieve performance improvement across QA benchmarks, many prioritize final answer correctness while overlooking the quality of intermediate reasoning steps, which may lead to chain-of-thought unfaithfulness. In this paper, we first introduce a comprehensive evaluation framework for evaluating RL-based search agents, covering three distinct faithfulness metrics: information-think faithfulness, think-answer faithfulness, and think-search faithfulness. Our evaluations reveal that a prototypical RL-based search agent, Search-R1, has significant room for improvement in this regard. To foster faithful reasoning, we introduce VERITAS (Verifying Entailed Reasoning through Intermediate Traceability in Agentic Search), a novel framework that integrates fine-grained faithfulness rewards into the reinforcement learning process. Our experiments show that models trained with VERITAS not only significantly improve reasoning faithfulness, but also achieve comparable task performance across seven QA benchmarks.

Figures (4)

• Figure 1: Evaluation of Search-R1. $^*$ and $^\dagger$ denotes LLMaaJ and trained reward model, respectively. We can see models trained with outcome-based reward, their good task performance does not automatically guarantee fully faithful reasoning chains, which may be suboptimal for practical deployment of such systems.
• Figure 2: The pipeline of VERITAS-R1 in the RL framework with PPO RL algorithm. Applying process supervision (VERITAS Reward Model) improves the faithfulness of policy model's intermediate reasoning steps. Outcome in this figure refers to the rollout trajectory used for subsequent reward calculation and advantage estimation.
• Figure 3: Faithfulness evaluation results comparing two variants of Search-R1 and our methods.
• Figure 4: Hyperparameter sensitivity study.