Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn LLM Agents

TL;DR

IGPO addresses reward-sparsity in multi-turn LLM agents by introducing turn-level information gain rewards that track the incremental increase in the ground-truth probability, combined with final outcome rewards. It provides dense, ground-truth-aware supervision without external reward models or high-variance Monte Carlo estimation, utilizes group-relative policy optimization with turn-level discounted advantages, and demonstrates improved accuracy and sample efficiency on both in-domain and out-of-domain benchmarks, especially for smaller models. The approach mitigates advantage collapse and improves credit assignment in long-horizon reasoning tasks, offering a practical and scalable enhancement for agentic RL in search-based settings.

Abstract

Large language model (LLM)-based agents are increasingly trained with reinforcement learning (RL) to enhance their ability to interact with external environments through tool use, particularly in search-based settings that require multi-turn reasoning and knowledge acquisition. However, existing approaches typically rely on outcome-based rewards that are only provided at the final answer. This reward sparsity becomes particularly problematic in multi-turn settings, where long trajectories exacerbate two critical issues: (i) advantage collapse, where all rollouts receive identical rewards and provide no useful learning signals, and (ii) lack of fine-grained credit assignment, where dependencies between turns are obscured, especially in long-horizon tasks. In this paper, we propose Information Gain-based Policy Optimization (IGPO), a simple yet effective RL framework that provides dense and intrinsic supervision for multi-turn agent training. IGPO models each interaction turn as an incremental process of acquiring information about the ground truth, and defines turn-level rewards as the marginal increase in the policy's probability of producing the correct answer. Unlike prior process-level reward approaches that depend on external reward models or costly Monte Carlo estimation, IGPO derives intrinsic rewards directly from the model's own belief updates. These intrinsic turn-level rewards are combined with outcome-level supervision to form dense reward trajectories. Extensive experiments on both in-domain and out-of-domain benchmarks demonstrate that IGPO consistently outperforms strong baselines in multi-turn scenarios, achieving higher accuracy and improved sample efficiency.

Figures (8)

• Figure 1: Proportion of zero-advantage groups during training—IGPO vs. GRPO on Qwen2.5-7B/3B-Instruct.
• Figure 2: The training pipeline of IGPO. (Upper) Turn-level information gain rewards are computed by measuring changes in ground-truth probability and combined with the outcome reward to derive discounted advantages. (Lower) Each rollout contains at most $T-1$ interaction turns, where each turn includes a reasoning step, a tool call, and the returned tool response, followed by a final answer turn. During optimization, the loss on tool response is masked out.
• Figure 3: Training curves on Qwen2.5-7B-Instruct with different reward designs.
• Figure 4: Mean reduction in ground-truth answer entropy from the initial query (Turn 0) to the last non-answer turn ($T\!-\!1$) during training.
• Figure 5: Token Efficiency: average performance with respect to the number of tokens used for gradient updates.

Theorems & Definitions (7)

• Definition A.1: Snowball Error in Multi-turn Agentic RL
• Lemma A.2: Lower bound of error rate
• proof : Proof Sketch
• Remark
• Theorem A.4: Process Reward as a Bound on Snowball Error
• proof
• proof