TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimization

TL;DR

This work identifies a Double Homogenization Dilemma in multi-turn tool-augmented reasoning, where sparse outcome rewards erase intermediate progress and homogenize gradients within trajectory groups. It presents Turn-level Stage-aware Policy Optimization (TSPO) which uses First-Occurrence Latent Reward (FOLR) to assign turn-level rewards at the earliest appearance of the ground-truth, preserving process signals and increasing within-group variance. TSPO replaces trajectory-level GRPO signals with per-turn advantages calculated through group-normalized rewards, enabling finer-grained learning without external annotations. Across seven QA benchmarks and two model scales, TSPO achieves substantial accuracy gains over state-of-the-art baselines while maintaining comparable training costs, demonstrating improved reward signaling and more stable optimization. The approach offers a lightweight, generalizable pathway to more sample-efficient, interpretable, and robust multi-turn reasoning with external tools.

Abstract

Multi-turn tool-integrated reasoning enables Large Language Models (LLMs) to solve complex tasks through iterative information retrieval. However, current reinforcement learning (RL) frameworks for search-augmented reasoning predominantly rely on sparse outcome-level rewards, leading to a "Double Homogenization Dilemma." This manifests as (1) Process homogenization, where the thinking, reasoning, and tooling involved in generation are ignored. (2) Intra-group homogenization, coarse-grained outcome rewards often lead to inefficiencies in intra-group advantage estimation with methods like Group Relative Policy Optimization (GRPO) during sampling. To address this, we propose Turn-level Stage-aware Policy Optimization (TSPO). TSPO introduces the First-Occurrence Latent Reward (FOLR) mechanism, allocating partial rewards to the step where the ground-truth answer first appears, thereby preserving process-level signals and increasing reward variance within groups without requiring external reward models or any annotations. Extensive experiments demonstrate that TSPO significantly outperforms state-of-the-art baselines, achieving average performance gains of 24% and 13.6% on Qwen2.5-3B and 7B models, respectively.

Figures (7)

• Figure 1: Reasoning trajectory distribution ($O$ vs. $P$) for Qwen2.5-7B-Instruct: no retrieval-free successes ($O^{+}/P^{-}$); near-miss ($O^{-}/P^{+}$) and total-failure cases ($O^{-}/P^{-}$ ) treated equally.
• Figure 2: Trajectory groups with rollout size $G=5$, composed by all-correct, mixed and all-wrong, during training on Qwen2.5-7B-Instruct.
• Figure 3: Overview of TSPO. Unlike outcome-level RL which assigns identical zero rewards to $O_2$ ($O^-/P^+$, middle) and $O_G$ ($O^-/P^-$, bottom), TSPO identifies the first occurrence ($t^*$, red key) of $a_{\mathrm{gold}}$ to assign turn-level rewards and performs advantage estimation on a per-turn basis. This restores intra-group variance and provides fine-grained signals. A detailed walkthrough of this example is provided in Section \ref{['par:example']}.
• Figure 4: Comparison of Training Dynamics. Left: Policy entropy collapses to near zero in GRPO but remains stable under TSPO variants, preserving reasoning diversity. Middle: KL divergence from GRPO spikes in the baseline, indicating unstable policy drift; TSPO maintains consistent alignment. Right: Gradient norms are large and volatile in GRPO due to sparse rewards, while TSPO yields smoother, more consistent updates.
• Figure 5: Training reward curves on four representative datasets. TSPO variants show consistently faster and more stable reward convergence compared to the GRPO baseline.