Training Multi-Turn Search Agent via Contrastive Dynamic Branch Sampling

TL;DR

BranPO introduces a tail-focused, contrastive supervision framework for long-horizon agentic RL, addressing credit assignment ambiguity by recursively truncating trajectories from the tail and resampling contrastive suffixes. By coupling difficulty-aware branch sampling with Redundant Step Masking, BranPO concentrates exploration on informative tail decisions while suppressing uninformative continuations, yielding stable, sample-efficient updates. The approach unifies stable prefix gradients (GRPO-like) with DPO-like suffix optimization, improving performance on long-horizon multi-hop QA benchmarks without increasing training budget. Empirical results show BranPO consistently outperforms strong baselines across seven QA datasets and generalizes to web-search scenarios, highlighting its potential for scalable, efficient agentic reasoning. Limitations remain in ultra-long horizons and very hard questions, pointing to future work on data quality and broader tool-use contexts.

Abstract

Agentic reinforcement learning has enabled large language models to perform complex multi-turn planning and tool use. However, learning in long-horizon settings remains challenging due to sparse, trajectory-level outcome rewards. While prior tree-based methods attempt to mitigate this issue, they often suffer from high variance and computational inefficiency. Through empirical analysis of search agents, We identify a common pattern: performance diverges mainly due to decisions near the tail. Motivated by this observation, we propose Branching Relative Policy Optimization (BranPO), a value-free method that provides step-level contrastive supervision without dense rewards. BranPO truncates trajectories near the tail and resamples alternative continuations to construct contrastive suffixes over shared prefixes, reducing credit ambiguity in long-horizon rollouts. To further boost efficiency and stabilize training, we introduce difficulty-aware branch sampling to adapt branching frequency across tasks, and redundant step masking to suppress uninformative actions. Extensive experiments on various question answering benchmarks demonstrate that BranPO consistently outperforms strong baselines, achieving significant accuracy gains on long-horizon tasks without increasing the overall training budget. Our code is available at \href{https://github.com/YubaoZhao/BranPO}{code}.

Figures (5)

• Figure 1: Comparison between GRPO, tree-based GRPO, and BranPO. Yellow nodes denote intermediate steps; green and red nodes indicate correct and incorrect answers. GRPO samples from the trajectory start, which is inefficient because SFT-trained models tend to produce highly similar prefixes. Tree-based GRPO branches from intermediate steps; however, more precise credit assignment demands a larger sampling tree and higher training overhead, and when branches are indistinguishable, they cannot provide effective supervision signals. BranPO samples from the trajectory end and enforces contrastive branches with a redundancy mask.
• Figure 2: Overview of BranPO. After the initial rollout, group accuracy is computed and branching budgets are assigned based on task accuracy and trajectory reward. Simple branching is applied to correct trajectories in easy tasks, while recursive branching is used for hard tasks or incorrect trajectories. Redundancy checks are performed on inefficient trajectories within easy tasks. Within a task, shared prefixes from all trajectories form a base group, while branched continuations from all trajectories constitute a branch group for relative advantage computation, with advantages corresponding to inefficient steps masked.
• Figure 3: Pass@K results of SFT-trained Qwen2.5 and Qwen3 models with partial resampling at the trajectory tail.
• Figure 4: Training dynamics on Qwen2.5-7B-Instruct, including the training rewards, the change in the average number of search steps per trajectory, the proportion of trajectories that contain branches with different rewards, and the proportion of trajectories that are detected to include redundant steps under BranPO.
• Figure 5: Training dynamics when scaling the number of training interactions to 8, showing the reward and the average number of search steps during training for GRPO, BranPO, and BranPO without RSM on Qwen3-4B-Instruct. RSM effectively suppresses unnecessary growth in search steps.

Theorems & Definitions (8)

• Theorem 2.1: Reduction to GRPO
• proof
• Lemma 2.2: Unbiased prefix value estimate without filtering
• proof
• Lemma 2.3: Conditional variance reduction
• proof
• Lemma 2.4: Range compression under contrastive filtering
• proof