AT$^2$PO: Agentic Turn-based Policy Optimization via Tree Search

TL;DR

AT$^2$PO addresses three core issues in multi-turn agentic RL: limited exploration under budget constraints, sparse turn-wise credit signals, and misalignment between turn-based decision making and flat optimization. It unifies three components—Entropy-Guided Tree Expansion for strategic turn-level exploration, Turn-wise Credit Assignment for fine-grained reward propagation, and Agentic Turn-based Policy Optimization for turn-aligned learning—to improve rollout diversity, learning efficiency, and stability. Empirical results across seven benchmarks show consistent gains over strong baselines, with particularly large improvements on multi-hop tasks, and ablation studies confirm the contribution of each component. The framework is orthogonal to tree search and can be integrated into existing multi-turn RL pipelines, offering a practical path to more robust agentic reasoning and tool use in LLM agents.

Abstract

LLM agents have emerged as powerful systems for tackling multi-turn tasks by interleaving internal reasoning and external tool interactions. Agentic Reinforcement Learning has recently drawn significant research attention as a critical post-training paradigm to further refine these capabilities. In this paper, we present AT$^2$PO (Agentic Turn-based Policy Optimization via Tree Search), a unified framework for multi-turn agentic RL that addresses three core challenges: limited exploration diversity, sparse credit assignment, and misaligned policy optimization. AT$^2$PO introduces a turn-level tree structure that jointly enables Entropy-Guided Tree Expansion for strategic exploration and Turn-wise Credit Assignment for fine-grained reward propagation from sparse outcomes. Complementing this, we propose Agentic Turn-based Policy Optimization, a turn-level learning objective that aligns policy updates with the natural decision granularity of agentic interactions. ATPO is orthogonal to tree search and can be readily integrated into any multi-turn RL pipeline. Experiments across seven benchmarks demonstrate consistent improvements over the state-of-the-art baseline by up to 1.84 percentage points in average, with ablation studies validating the effectiveness of each component. Our code is available at https://github.com/zzfoutofspace/ATPO.

Figures (8)

• Figure 1: Above: Token-level entropy distribution in a search agentic rollout sample. A complete rollout sequence is sampled from multiple turns. Below: Turn-wise entropy box plot analysis. Substantial discrepancies exist among turn-level token groups.
• Figure 2: Overview of the AT2PO framework, including entropy-guided tree-structured rollout, turn-wise credit assignment for fine-grained supervision and turn-based policy optimization during reinforcement training.
• Figure 3: Experiment results on three backbone models across seven datasets. The bolded values indicate the best result in comparisons. Our AT2PO outperforms existing methods in the majority of cases.
• Figure 5: Turn count distribution per sample in the validation set across Multi-hop and Single-hop benchmarks with Qwen3-4B fine-tuned using AT2PO.
• Figure 6: $H_{turn}$ alongside training steps of AT2PO in all experiment settings.