Agentic Entropy-Balanced Policy Optimization

TL;DR

This work identifies two entropy-driven challenges in agentic reinforcement learning for web agents: high-entropy rollout collapse and high-entropy token gradient clipping. It proposes AEPO, comprising a dynamic entropy-balanced rollout with entropy pre-monitoring and branch penalties, and entropy-balanced policy optimization that uses a stop-gradient in clipping and entropy-aware advantage estimation. Across 14 challenging benchmarks, AEPO consistently outperforms mainstream RL baselines, achieving strong results even with limited training samples (1k) and improving rollout diversity while maintaining stable entropy. The findings offer a scalable, robust approach for training general web agents with long-horizon tool usage and uncertainty-aware learning dynamics.

Abstract

Recently, Agentic Reinforcement Learning (Agentic RL) has made significant progress in incentivizing the multi-turn, long-horizon tool-use capabilities of web agents. While mainstream agentic RL algorithms autonomously explore high-uncertainty tool-call steps under the guidance of entropy, excessive reliance on entropy signals can impose further constraints, leading to the training collapse. In this paper, we delve into the challenges caused by entropy and propose the Agentic Entropy-Balanced Policy Optimization (AEPO), an agentic RL algorithm designed to balance entropy in both the rollout and policy update phases. AEPO comprises two core components: (1) a dynamic entropy-balanced rollout mechanism that adaptively allocate global and branch sampling budget through entropy pre-monitoring, while imposing a branch penalty on consecutive high-entropy tool-call steps to prevent over-branching issues; and (2) Entropy-Balanced Policy Optimization that inserts a stop-gradient operation into the high-entropy clipping term to preserve and properly rescale gradients on high-entropy tokens, while incorporating entropy-aware advantage estimation to prioritize learning on high-uncertainty tokens. Results across 14 challenging datasets show that AEPO consistently outperforms 7 mainstream RL algorithms. With just 1K RL samples, Qwen3-14B with AEPO achieves impressive results: 47.6% on GAIA, 11.2% on Humanity's Last Exam, and 43.0% on WebWalker for Pass@1; 65.0% on GAIA, 26.0% on Humanity's Last Exam, and 70.0% on WebWalker for Pass@5. Further analysis reveals that AEPO improves rollout sampling diversity while maintaining stable policy entropy, facilitating scalable web agent training.

Figures (8)

• Figure 1: Performance overview of AEPO algorithm.
• Figure 2: Two high-entropy challenges in agentic RL. (1) High-Entropy Rollout Collapse: Over-branching at high-entropy steps along specific paths, limiting exploration of other potential correct branches; (2) High-Entropy Token Gradient Clipping: Consistent clipping of high-entropy token gradients during policy updates hinders learning effective exploration behaviors.
• Figure 3: Quantitative statistics of two entropy-based challenges in web agent RL training.
• Figure 4: The overview of Agentic Entropy-Balanced Policy Optimization.
• Figure 5: The comparison analysis of Qwen3-14B using ARPO and AEPO across Pass@1 to Pass@5 metrics.