Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agents

Deyu Zou, Yongqiang Chen, Fan Feng, Mufei Li, Pan Li, Yu Gong, James Cheng

Abstract

Reinforcement learning (RL) with outcome-based rewards has achieved significant success in training large language model (LLM) agents for complex reasoning tasks. However, in active reasoning where agents need to strategically ask questions to acquire task-relevant information, we find that LLM agents trained with RL often suffer from information self-locking: the agent ceases to ask informative questions and struggles to internalize already-obtained information. To understand the phenomenon, we decompose active reasoning into two core capabilities: Action Selection (AS), which determines the observation stream through queries, and Belief Tracking (BT), which updates the agent's belief based on collected evidence. We show that deficient AS and BT capabilities will limit the information exploration during RL training. Furthermore, insufficient exploration in turn hinders the improvement of AS and BT, creating a feedback loop that locks the agent in a low-information regime. To resolve the issue, we propose a simple yet effective approach that reallocates the learning signal by injecting easy- to-obtain directional critiques to help the agent escape self-locking. Extensive experiments with 7 datasets show that our approach significantly mitigates the information self-locking, bringing up to 60% improvements.

On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agents

Abstract

Reinforcement learning (RL) with outcome-based rewards has achieved significant success in training large language model (LLM) agents for complex reasoning tasks. However, in active reasoning where agents need to strategically ask questions to acquire task-relevant information, we find that LLM agents trained with RL often suffer from information self-locking: the agent ceases to ask informative questions and struggles to internalize already-obtained information. To understand the phenomenon, we decompose active reasoning into two core capabilities: Action Selection (AS), which determines the observation stream through queries, and Belief Tracking (BT), which updates the agent's belief based on collected evidence. We show that deficient AS and BT capabilities will limit the information exploration during RL training. Furthermore, insufficient exploration in turn hinders the improvement of AS and BT, creating a feedback loop that locks the agent in a low-information regime. To resolve the issue, we propose a simple yet effective approach that reallocates the learning signal by injecting easy- to-obtain directional critiques to help the agent escape self-locking. Extensive experiments with 7 datasets show that our approach significantly mitigates the information self-locking, bringing up to 60% improvements.
Paper Structure (35 sections, 9 theorems, 111 equations, 8 figures, 2 tables)

This paper contains 35 sections, 9 theorems, 111 equations, 8 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Self-Locking in Active Reasoning
  3. Preliminaries
  4. Testbeds and decomposed proxies
  5. Failure modes in reinforcement learning training
  6. Understanding Self-Locking
  7. Breaking Self-Locking with Directions
  8. Stepwise directional critiques
  9. Injecting directional critiques into policy-gradient
  10. Theoretical and Practical Discussion
  11. Experiments
  12. Experimental Setup
  13. Experimental Results and Analyses
  14. Conclusion
  15. Related Work
  16. ...and 20 more sections

Key Result

Theorem 3.4

Fix $\delta,\varepsilon>0$, assume (i) LR-Lipschitz reward-belief updates, (ii) updates to $\mathcal{I}_{\mathrm{th}}(\omega)$ and $C_{\mathrm{BT}}(\omega)$ are bounded by some constant $G<\infty$, (iii) self-destructive drift to belief is invariant to the query choice, (iv) absorbed belief update i where $\preceq$ denotes elementwise inequality, $\beta_I=2(H+1)GL_R\kappa_U$, $\beta_C=2(H+1)GL_R$

Figures (8)

  • Figure 1: Overall illustration of information self-locking (SeL) and its mitigation. $(b,a,o)$ denote the agent’s internal belief, its chosen action, and the resulting feedback at each turn. Under vanilla outcome-based RL (top), the agent can become trapped in a self-locking regime: deficient belief tracking masks contributions of informative queries, leading to misaligned credit assignment. Our AReW (bottom) introduces advantage reweighting via directional critiques, correcting the learning signal and assisting to mitigate SeL in active reasoning.
  • Figure 2: (a)/(b): the training dynamics of outcome reward, per-turn AS, and per-turn BT proxies in PE-G$_{S=2}$ and MediQ datasets (Qwen-2.5-7B-Instruct). (c)/(d): correlation between the reward and AS proxies in PE-G$_{S=2}$ and MediQ (Qwen-2.5-7B-Instruct). In strong BT patterns (by human-defined rules or frontier LLMs), the same AS sequence exhibits stronger correlation with the final reward.
  • Figure 3: Training dynamics of rewards, evaluated under the PPO algorithm with Qwen-2.5-7B-Instruct across vanilla PPO, PPO with AReW -- as-only and PPO with AReW -- as+bt.
  • Figure 4: Evaluations of AS and BT capabilities under PPO algorithm with Qwen-2.5-7B-Instruct across vanilla PPO, PPO with AReW.
  • Figure 5: (a): outcome-RL reduces sensitivity to interactive feedback while increasing belief consistency. (b)/(c): Evaluations of AS and BT capabilities under GRPO and GSPO (Qwen-2.5-7B-Instruct). (d): Training dynamics of rewards under different strength of AReW.
  • ...and 3 more figures

Theorems & Definitions (24)

  • Definition 3.1: AS Informativeness
  • Definition 3.2: Belief drift
  • Definition 3.3: Self-Locking Regime
  • Theorem 3.4: Informal
  • Proposition 4.1
  • Definition 2.2
  • Definition 2.3
  • Definition 2.7: Locking regime
  • Proposition 2.10: AS-channel outcome advantages
  • Proposition 2.11: BT-channel outcome advantages
  • ...and 14 more