Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimizing Agentic Reasoning with Retrieval via Synthetic Semantic Information Gain Reward

Senkang Hu, Yong Dai, Yuzhi Zhao, Yihang Tao, Yu Guo, Zhengru Fang, Sam Tak Wu Kwong, Yuguang Fang

TL;DR

InfoReasoner advances agentic reasoning by redefining information gain as uncertainty reduction over belief states and implementing a scalable, output-aware intrinsic reward derived from semantic clustering of model outputs. The approach provides formal guarantees (non-negativity, telescoping additivity, monotonicity) and uses bidirectional textual entailment to form semantic classes, enabling a dense information gain signal that guides retrieval. Empirically, it achieves state-of-the-art results across seven QA benchmarks at 3B and 7B scales, with up to 5.4% average accuracy gains and improved inference efficiency. The work offers a principled, scalable pathway to reliable retrieval-enhanced reasoning in large language models with practical reward design and GRPO-based optimization.

Abstract

Agentic reasoning enables large reasoning models (LRMs) to dynamically acquire external knowledge, but yet optimizing the retrieval process remains challenging due to the lack of dense, principled reward signals. In this paper, we introduce InfoReasoner, a unified framework that incentivizes effective information seeking via a synthetic semantic information gain reward. Theoretically, we redefine information gain as uncertainty reduction over the model's belief states, establishing guarantees, including non-negativity, telescoping additivity, and channel monotonicity. Practically, to enable scalable optimization without manual retrieval annotations, we propose an output-aware intrinsic estimator that computes information gain directly from the model's output distributions using semantic clustering via bidirectional textual entailment. This intrinsic reward guides the policy to maximize epistemic progress, enabling efficient training via Group Relative Policy Optimxization (GRPO). Experiments across seven question-answering benchmarks demonstrate that InfoReasoner consistently outperforms strong retrieval-augmented baselines, achieving up to 5.4% average accuracy improvement. Our work provides a theoretically grounded and scalable path toward agentic reasoning with retrieval.

Optimizing Agentic Reasoning with Retrieval via Synthetic Semantic Information Gain Reward

TL;DR

InfoReasoner advances agentic reasoning by redefining information gain as uncertainty reduction over belief states and implementing a scalable, output-aware intrinsic reward derived from semantic clustering of model outputs. The approach provides formal guarantees (non-negativity, telescoping additivity, monotonicity) and uses bidirectional textual entailment to form semantic classes, enabling a dense information gain signal that guides retrieval. Empirically, it achieves state-of-the-art results across seven QA benchmarks at 3B and 7B scales, with up to 5.4% average accuracy gains and improved inference efficiency. The work offers a principled, scalable pathway to reliable retrieval-enhanced reasoning in large language models with practical reward design and GRPO-based optimization.

Abstract

Agentic reasoning enables large reasoning models (LRMs) to dynamically acquire external knowledge, but yet optimizing the retrieval process remains challenging due to the lack of dense, principled reward signals. In this paper, we introduce InfoReasoner, a unified framework that incentivizes effective information seeking via a synthetic semantic information gain reward. Theoretically, we redefine information gain as uncertainty reduction over the model's belief states, establishing guarantees, including non-negativity, telescoping additivity, and channel monotonicity. Practically, to enable scalable optimization without manual retrieval annotations, we propose an output-aware intrinsic estimator that computes information gain directly from the model's output distributions using semantic clustering via bidirectional textual entailment. This intrinsic reward guides the policy to maximize epistemic progress, enabling efficient training via Group Relative Policy Optimxization (GRPO). Experiments across seven question-answering benchmarks demonstrate that InfoReasoner consistently outperforms strong retrieval-augmented baselines, achieving up to 5.4% average accuracy improvement. Our work provides a theoretically grounded and scalable path toward agentic reasoning with retrieval.
Paper Structure (40 sections, 3 theorems, 27 equations, 10 figures, 4 tables, 2 algorithms)

This paper contains 40 sections, 3 theorems, 27 equations, 10 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Agentic Reasoning
  4. Retrieval-Augmented Generation
  5. Reinforcement Learning for LLM Reasoning
  6. Rethinking Information Gain as Uncertainty Reduction over Belief States
  7. Bayesian Belief Update
  8. Uncertainty Functional
  9. Information Gain as Uncertainty Reduction
  10. Interpretation
  11. Method
  12. Semantic Clustering via Bidirectional Textual Entailment
  13. Synthesizing Information Gain Reward via Semantic Uncertainty Reduction
  14. Policy Optimization with Information Gain Reward
  15. Experiments
  16. ...and 25 more sections

Key Result

Proposition 3.6

If $U$ satisfies the Expected Non-Increase axiom described in Eq. eq:expected-monotonicity, then for any belief $b$ and action $a$, under the assumption of consistent Bayesian belief updates, the Expected Information Gain is non-negative: with equality iff $O \perp Y \mid (b,a)$. This theoretical lower bound serves as the optimality condition for our agentic reasoning policy.

Figures (10)

  • Figure 1: Overview of InfoReasoner. The framework estimates the agent's belief state by sampling candidate answers and grouping semantically equivalent ones. It then calculates an Information Gain intrinsic reward by measuring the reduction in semantic uncertainty (entropy) when retrieved evidence is provided compared to a retrieval-free baseline, thereby incentivizing the agent to acquire uncertainty-resolving information.
  • Figure 2: Training dynamics analysis: (a) EM score trajectories comparison between InfoReasoner and Search-R1; (b) Decomposition of total reward into Information Gain (IG) and EM scores; (c) Entropy loss comparison; (d) Response length comparison between InfoReasoner and Search-R1.
  • Figure 3: Case study of InfoReasoner.
  • Figure 4: Information gain analysis: (a) Comparison across different retrieval scenarios demonstrating synergistic effects when jointly observing multiple documents; (b) Sensitivity analysis of group size $M$ on estimation accuracy, showing the trade-off between computational efficiency and reward signal quality.
  • Figure 5: System prompt used for training and inference, guiding the model to perform iterative reasoning and retrieval.
  • ...and 5 more figures

Theorems & Definitions (12)

  • Definition 3.1: Belief State
  • Definition 3.2: Bayes Belief Update
  • Definition 3.3: Uncertainty Functional
  • Definition 3.4: One-step Information Gain
  • Definition 3.5: Expected Information Gain
  • Proposition 3.6: Non-negativity under Ideal Updates
  • Proposition 3.7: Telescoping Additivity
  • Proposition 3.8: Monotonicity w.r.t. Information Channels
  • Definition 4.1: Semantic Equivalence via Bidirectional Entailment
  • proof
  • ...and 2 more