Scaling Search-Augmented LLM Reasoning via Adaptive Information Control

TL;DR

The paper tackles the problem of uncontrolled information acquisition in search-augmented LLM reasoning, where excessive or poorly targeted retrieval harms performance and training stability. It introduces DeepControl, a framework that defines information utility as the marginal value of retrieved evidence and uses two core controls—granularity control via hierarchical selective expansion and retrieval continuation control—to regulate when and how much information is exposed. An annealed control strategy guides the agent from externally guided retrieval to autonomous, internally learned behavior, supported by a composite reward that balances answer accuracy, tool use discipline, and retrieval effectiveness. Empirical results across seven benchmarks and multiple model scales show consistent improvements over strong baselines, along with improved training stability and efficiency, demonstrating the value of treating information acquisition as a controllable, learnable process.

Abstract

Search-augmented reasoning agents interleave multi-step reasoning with external information retrieval, but uncontrolled retrieval often leads to redundant evidence, context saturation, and unstable learning. Existing approaches rely on outcome-based reinforcement learning (RL), which provides limited guidance for regulating information acquisition. We propose DeepControl, a framework for adaptive information control based on a formal notion of information utility, which measures the marginal value of retrieved evidence under a given reasoning state. Building on this utility, we introduce retrieval continuation and granularity control mechanisms that selectively regulate when to continue and stop retrieval, and how much information to expand. An annealed control strategy enables the agent to internalize effective information acquisition behaviors during training. Extensive experiments across seven benchmarks demonstrate that our method consistently outperforms strong baselines. In particular, our approach achieves average performance improvements of 9.4% and 8.6% on Qwen2.5-7B and Qwen2.5-3B, respectively, over strong outcome-based RL baselines, and consistently outperforms both retrieval-free and retrieval-based reasoning methods without explicit information control. These results highlight the importance of adaptive information control for scaling search-augmented reasoning agents to complex, real-world information environments.

Figures (6)

• Figure 1: Definition of a search step. A search step starts with a retrieval action and includes all subsequent expansion actions until the next retrieval or termination.
• Figure 2: Per-rollout information utility across search steps. Each curve corresponds to a rollout, illustrating how utility evolves with additional evidence.
• Figure 3: Hierarchical granularity control via selective expansion, where the agent incrementally refines retrieved information from coarse summaries to finer-grained content levels as needed.
• Figure 4: Trajectories generated in rollout mode with and without information control.
• Figure 5: (a) DeepControl vs. PPO: our approach reaches higher reward throughout training under the same optimization setup. (b) PPO vs. GRPO: GRPO leads to reward collapse, while PPO shows steadier optimization and maintains stable performance. (c) Response-length behavior during training: the average response length and training reward evolve together over time, with response length increasing at early stages and tending to stabilize later.

Theorems & Definitions (4)

• Lemma 1: Monotonicity
• proof
• Lemma 2: Diminishing Returns
• proof