Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarial Yet Cooperative: Multi-Perspective Reasoning in Retrieved-Augmented Language Models

Can Xu, Lingyong Yan, Jiayi Wu, Haosen Wang, Shuaiqiang Wang, Yuchen Li, Jizhou Huang, Dawei Yin, Xiang Li

TL;DR

The paper tackles reliability and interpretability in retrieved-augmented reasoning by introducing ARR, a multi-agent framework where a Reasoner and a Verifier engage in adversarial yet cooperative dialogue over retrieved evidence. It introduces two core rewards—Adversarial Outcome Rewards and a token-level Process-aware Advantage—that jointly promote final correctness and high-quality, evidence-grounded reasoning traces, with GRPO used for stable policy optimization. Empirical results across general and multi-hop QA benchmarks show ARR consistently outperforms strong baselines and even surpasses larger models on several tasks, illustrating the efficacy of multi-perspective reasoning and process supervision. The approach advances practical RAG systems by fostering rigorous debate, transparent verification, and uncertainty reduction, enabling more robust, scalable reasoning in real-world settings.

Abstract

Recent advances in synergizing large reasoning models (LRMs) with retrieval-augmented generation (RAG) have shown promising results, yet two critical challenges remain: (1) reasoning models typically operate from a single, unchallenged perspective, limiting their ability to conduct deep, self-correcting reasoning over external documents, and (2) existing training paradigms rely excessively on outcome-oriented rewards, which provide insufficient signal for shaping the complex, multi-step reasoning process. To address these issues, we propose an Reasoner-Verifier framework named Adversarial Reasoning RAG (ARR). The Reasoner and Verifier engage in reasoning on retrieved evidence and critiquing each other's logic while being guided by process-aware advantage that requires no external scoring model. This reward combines explicit observational signals with internal model uncertainty to jointly optimize reasoning fidelity and verification rigor. Experiments on multiple benchmarks demonstrate the effectiveness of our method.

Adversarial Yet Cooperative: Multi-Perspective Reasoning in Retrieved-Augmented Language Models

TL;DR

The paper tackles reliability and interpretability in retrieved-augmented reasoning by introducing ARR, a multi-agent framework where a Reasoner and a Verifier engage in adversarial yet cooperative dialogue over retrieved evidence. It introduces two core rewards—Adversarial Outcome Rewards and a token-level Process-aware Advantage—that jointly promote final correctness and high-quality, evidence-grounded reasoning traces, with GRPO used for stable policy optimization. Empirical results across general and multi-hop QA benchmarks show ARR consistently outperforms strong baselines and even surpasses larger models on several tasks, illustrating the efficacy of multi-perspective reasoning and process supervision. The approach advances practical RAG systems by fostering rigorous debate, transparent verification, and uncertainty reduction, enabling more robust, scalable reasoning in real-world settings.

Abstract

Recent advances in synergizing large reasoning models (LRMs) with retrieval-augmented generation (RAG) have shown promising results, yet two critical challenges remain: (1) reasoning models typically operate from a single, unchallenged perspective, limiting their ability to conduct deep, self-correcting reasoning over external documents, and (2) existing training paradigms rely excessively on outcome-oriented rewards, which provide insufficient signal for shaping the complex, multi-step reasoning process. To address these issues, we propose an Reasoner-Verifier framework named Adversarial Reasoning RAG (ARR). The Reasoner and Verifier engage in reasoning on retrieved evidence and critiquing each other's logic while being guided by process-aware advantage that requires no external scoring model. This reward combines explicit observational signals with internal model uncertainty to jointly optimize reasoning fidelity and verification rigor. Experiments on multiple benchmarks demonstrate the effectiveness of our method.
Paper Structure (29 sections, 1 theorem, 10 equations, 7 figures, 2 tables)

This paper contains 29 sections, 1 theorem, 10 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reward Design and Process Supervision
  4. LRMs Synergizied with RAG
  5. Preliminary
  6. Task Formulation
  7. Entropy Pattern Analysis
  8. Methods
  9. Multi-perspective Reasoning
  10. Multi-perspective Optimization
  11. Adversarial Outcome Rewards
  12. Process-aware Advantage
  13. (1) Answer Correctness
  14. (2) Verifier Clarity
  15. (3) Behavior Impact
  16. ...and 14 more sections

Key Result

Proposition 1

In an ideal agentic RAG system, as relevant information is retrieved, both the uncertainty of agent and the policy entropy monotonically decrease.

Figures (7)

  • Figure 1: Ideal agent certainty through iterative search and reasoning
  • Figure 2: Statistical analysis of policy entropy pattern in Search-R1 trajectories. The y-axis of the left subplots denotes the proportion of trajectories exhibiting specific pattern in all multi-turn ($\ge 3$) samples. The y-axis of the right subplots represents the average accuracy of samples grouped by their pattens.
  • Figure 3: Multi-perspective reasoning of ARR.
  • Figure 4: Process-aware advantage of ARR.
  • Figure 5: F1-score comparison.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Proposition 1
  • proof