$\mathbf{T^3}$: Reducing Belief Deviation in Reinforcement Learning for Active Reasoning

TL;DR

This work proposes to track the deviation of model beliefs and develop a simple yet effective method that detects excessive belief deviation and truncates trajectories during training to remove uninformative tails, and systematically improves policy optimization.

Abstract

Active reasoning requires large language models (LLMs) to interact with external sources and strategically gather information to solve problems. Central to this process is belief tracking: maintaining a coherent understanding of the problem state and the missing information toward the solution. However, due to limited reasoning capabilities, LLM-based agents often suffer from belief deviation: they struggle to correctly model beliefs, lose track of problem states, and fall into uninformative or repetitive actions. Once this happens, errors compound and reinforcement learning (RL) training fails to properly credit the crucial exploratory steps. To address this issue, we propose to track the deviation of model beliefs and develop $\mathbf{T^3}$, a simple yet effective method that detects excessive belief deviation and truncates trajectories during training to remove uninformative tails. By preserving credit for informative prefixes, $\mathbf{T^3}$ systematically improves policy optimization. Across 5 challenging tasks, $\mathbf{T^3}$ consistently enhances training stability, token efficiency, and final performance, achieving up to 30% gains while cutting rollout tokens by roughly 25%. These results highlight belief control as a key principle for developing robust and generalizable LLM-based active reasoners.

Figures (11)

• Figure 1: Overall framework of T$^\mathbf{3}$. By truncating belief-trapped trajectories, we prevent the agent from entering the belief-trap region (BTR) where credit assignment is contaminated and becomes misleading, allowing learning signals to concentrate on genuinely informative actions. As a result, policy optimization becomes more stable and effective under complex active-reasoning environments.
• Figure 2: Training dynamics of rewards.
• Figure 3: Training dynamics of response length.
• Figure 4: Training dynamics of the ratio of early truncation w.r.t. steps under different truncation conditions for the SP (a), CD (b), and PE (c) tasks.
• Figure 5: Effectiveness of T$^\mathbf{3}$ on different sizes (a, b) and types (c) of LLM architectures. The "Performance Gain" denotes the improvement of T$^\mathbf{3}$ compared to the vanilla RL method.

Theorems & Definitions (20)

• Theorem 1: Informal
• Definition 1: Belief Trap Region, BTR
• Theorem 2: Informal
• Corollary 1: Value of Truncation
• Definition 2: T$^\mathbf{3}$ Condition
• Definition 3: Truth-anchored potential
• Definition 4: One-Step Informativeness
• Definition 5: One-step Agent Progress
• Definition 6: Agent-Bayes update error
• Definition 7: Belief Trap Region, BTR